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<54> Dsto SamSer via a cache under control of an independent message network 

<™ A svstem interface includes a plurality of first directors 180, a plurality of second directors 200 a data 
' * tl^Z ™*triam&m* network 260. The data transfer section includes a cache memory 220 which 
XTB Z S^i^rSi^wS ThTmLsaging network 260 operates independently of the data transfer 
secuon and is T^^^Snm and second directors 180, 200. Data transfer between the first doctors 
and the second SrSuSs is controlled in response to messages passing between the first ^^^ ln 
^nd dimis through the messaging network 260 and the data passes through the cache memory 220 m 

thG ^"^n^^ directors may be coupled respectively to computer/server 120 and a bank of disc 
drives 140. 
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At least one drawing 



riginally filed was informal and the print reproduc d here is taken from a later filed formal copy. 
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DATA STORAGE SYSTEM 

BACKGROUND OF THE IN VENTION 
This invention relates generally to data storage systems, and more particularly to 
data storage systems having redundancy arrangements to protect against total system failure in 
the event of a failure m a component or subassembly of the storage system. 

As is known in the art, large host computers and servers (coUectively referred to 
herein as "host computer/servers") require large capacity data storage systems. These large 
computer/servers generally includes data processors, which perform many operations on data 
fctroduced to the host computer/server through peripherals including the data storage system. 
The results of these operations are output to peripherals, including the storage system. 

Onetypeofdatastoragesystemisamagneticdiskstoragesystem. Hereabankof 

disk drives and the host computer/server are coupled together through an interface. The 
interface includes "front end" or host computer/server controllers (or directors) and "back-end" 
or disk controllers (or 'directors). The interface operates the controllers (or directors) in such a 
way that they are transparent to the host computer/server. That is, data is stored m, and 
retrieved from, the bank of disk drives in such a way that the host computer/server merely thmks 
itisoperatmgwimitsownlocaldiskdrive. One such system is described in U.S. Patent 
5^06939 enuued "System and Memod?^^^ 

Moshe Yanai, Natm Vishh^Bmno ^t^^D^el Castel, issued Aprils, 1993, and 

assigned to the same assignee as the present invention: ; 

As described in such U.S. Patent, the interface may also include, in addition to the. 
host computer/server controllers (or directo«)^^nttoUers.(or directors), addressable 
cache memories. The cache memory is a serrdconductor memory and is provided to rapxdly 
store data from the host computer/server, before storage in the disk drives, and, on the other 
hand, store data from the disk drives' prior to beinf Sent to the host computer/server. The cache 
memory being a semiconductor memory; as-distmgmshedW a magnetic memory as m the 
case of the disk drives, is much faster than me disk drives in reading and writing data. 



* The host computer/server con^oUers, disk controllers and cac 
interconnected through a backplane printed circuit board. More particularly, disk controllers are 
mounted on disk controller printed circuit boards. The host qpmputer/server controllers are 
mounted on host computer/server controller printed circuit boards. And, cache memories are 
mounted on cache memory printed circuit boards. The disk directors, host computer/server 
directors, and cache memory printed circuit boards, plug into the backplane printed circuit board. 
In order to provide data integrity in case of a failure in a . director, the backplane printed circuit 
board has; a pair of buses. One set of the disk directors r is connected to one bus and another set of 
the disk directors is connected to the other bus. Likewise, one, set of the host computer/server 

directors is connected to one bus and another set of the host computer/server directors is 
directors connected to the other bus. The cache memories are connected to both buses. Each 
one of the buses provides data, address and control information." " ~ _ r 

The arrangement is shown schematically in FIG. 1. Thus, the use of two buses Bl, 
B2 provides a degree of redundancy to protect against a total system failure in the event that 
the controllers or disk drives connected to one bus, fail. Further, the use of two buses 
increases the data transfer bandwidth of the system compared to a system having a single 
bus. Thus, in operation when the host computer/server 12 wishes to store data, the host 
computer 12 issues a write request to one of the front-end directors 14 (i.e., host 
computer/server directors) to perform awrite command. One of the front-end directors 14 
replies to the request and asks the host qpmputer 12 for the data. After the request has passed 
to the requesting one of the fronfcend directors 14 ? the director 14 determines the size of the 
data and reserves space in the cacheL memory 18 to store the request. Tlie front-end director 
14 then produces contrpl signals on one of the address memo^ busses Bl, B2 connected to 
such front-end director 14 to enable. Jhe r fra^fer to the cache memory 18. The host 
computer/server 12 then transfers the data to the front : end director 14. The front-end 
director 14 then advises the host cpmputer/server 12 that the transfer is complete. Tiie front- 
end director 14 looks up in a Tabl$, not : .shown,,stored : in the cache memory 18 to determine 
which one of the back-end directors 20 (i.e., disk directors) is to handle this request. The 
Table maps the host computer/server 12 addresses into an address in the bank 14 of disk 
drives. The front-end director 14 then puts a notification in a "mail box" (not shown and 
stored in the cache memory 18) for the back-end director 20, which is to handle the request, 



10 



15 



20 
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me amount of .He data and the disk address for the dat* Other back-end directors 20 po.l the 
cache memory 1 8 when «hey are idle to check iheir "maii boxes". If ft. polled "mail box- 
tadica.es a transfer is to be made, ihe back-end director 20 processes the request, addresses 
the disk drive in the bank 22, «ads the dam torn te cache memory 18 and writes it into the 
addresses of a disk drive in the bank 22. 

When data is to be read from a disk drive in bank 22 to the host computer/server 
12 the system operates in a reciprocal manner. More particularly, during a read operation, a 
read request is instituted by the host computer/server 12 for data at specified memory 
locations (U., a requested data block). One of me front-end directors 14 receives the read 
request and examines the cache memory 18 to determine whether the requested d*a block „ . , 
stored in the cache memory 18. If the requested datahlock is in the cache memory 18, me 
requested datablock is read from me cache memory 18 and is sent to the host, 
computer/server 12. Ifthe front-end director 14 determines ma. the requested data block ,s 
no , i„ me cache memory 18 a so-caUed "cache miss") and the director 14 writes a note 
m me cache memory 18 (i.e., the -mail box") that i, needs to receive the requested data 
block The back-end directors 20 poll the cache memory IS to determine whethe,there a an 
action to be taken (i.e., a read operation of the requested block of data). The one of the back- 
ed directors 20 which poU me cache meinoty 1 8 mail box and detects a read operation reads 
me requested dau block m d Mtia.es ^ ^ 

memory 18. ^**^*o^im^^^'^*" t * ^ /. 
complete indication is placed in me — I***-* to""** 

me front-end directors 14 are ponn^ 

' Wheaineof^ 

end director 14 completes the ^(*ottei*^**^^^"*° »** 
memory 18 to me host computer/server 12.' -"■■'-•"<■'- '--^ ■ 

The use of mailboxes and polling requires time to transfer data between the host 

•"; computer/server .2 and me bank" 22 *&*~^?^*'»** tt ** U ^ " 
the interface. 
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STTM |MARV O 'THf, IMVF.NTION 
The present invention provides a system interface as defined inclaim.l. a data storage 
system as defined in claim 8 and/or methods as defined in claims 15. 16 and/or 17. 
fc accordance with a first aspect of the present invention, a system interface is provided. Such 

.. r *T~T.' , nluralitv of second directors, a data transfer 
interface includes a plurality of first directors, a plurality or . - 

secdonandamesaagenetworic. The dau transfer section includes a cache memory. The 
cache memory is coupled to the plurality of firs, and second directors, The messagm* 
network operas independently of the data rransfer^onaud such r,^o*« coupled to 
fc^'^**--**^**""" 1 -** The firs, and second 

ZJil^&ti&i^*^**'^?"*-- **** 

nasses through tie cache memoryinthe data transfer secnon. 

operative independent of the data transfer section, for such messafmg thereby mcreasxng the 

ODOThM bandwidth ofthe system interlace. , . 

in one embodiment of me invenrion,rhe system interface each one of me firs, 
direcWs includes adatapipe coupledberween an input of such one of me firs, directors and 
^cachememory and a controller for Werrmgte messages be^een me message 

' h> one embodimeut each one of me second directors .includes a dataptpe coup led 

l " betweeVaimput'of snch one of me -ond dire^^and u,, cache metnory and a confer 

one embodiment the cUe-.ors includes; a data pipe coupled between an input of 
such one of me first directors and me cache memory^a rmcrpprocessor; and a controller 
Sled to the microprocessor-andthe datapipe ^controlling » 

between the ,s provided; 

* ' In accordance with another aspect of the. invention, a data si g 

• for transferring data between a host computer/server and a ban, of disk drives through a 
system interface. The system interface includes a plurality of first directors coupled to host 
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con.pu.er/scrve, a plurality of second directors coupled to mebankof d*k dnves a da* 
ttansfer section, and a message network dau transfer section inciudes a cache memory. 
The cache memory is coupled to the plurality o^^ 

network is operative independently of die data transfer section and such network ts coupled 
„ fte piurality' of firs, directors and the plurality of second directors. The firs, and second 
sectors c «^-dtt^»-«^W«"^^ rf ^.* l r ,B . ili 

,„ accordance with ye, another -encion. a method is provided for operant* a 

data storage system adapted to transfer data between a host computer/server and abank of 
disk^ves. The method includes ^ferrmg messages through a mes^g^network w, tt 
fcedatabeing transferred between the host computer/server and me bank of dtsk dnves 
tooughaeachememory.such message network being independent of me cache memory. 

In accordance with another aspect of the invention, a n,e<hod is provided for operaung a data 
storage system adapted to transfer da* between a host computer/server and a bank of drsk 
Ive! tough a system interface. 

and a dam transfer section having a cache memory, such cache memory b«ng coup W to ft. 
L W computer/server and me bank of disk «ves u^o^l pf fte first and ^nd 

''^nghamessagmgnerwo^ 

section, such message network being independent of the cache memory. 

BPTFFf>eSCT ^"^«FTHF, DRAWINGS 

lowing detailed j^^lAM******* accompanying drawmgs, » 



which: 



HO. 2 is a block diagram of a datastorage system according to the invention; 



FIG. 2A shows the fields of a descriptor used in the system interface of the data 

storage system of FIG. 2; ' ■: / ir- \ .. . 

FIG. 2B shows the filed used in a MAC packet used in the system interface of the 

data storage system of FIG, 2; ' 
5 - FIG. 3 is a sketch of an electrical cabinet storing a system interface used in the data 

storage system of FIG. 2; 

FIG. 4 is a diagramatical, isometric sketch showing printed circuit boards providing 

the system interface of the data storage system QfEIG.2; , . 

FIG. 5 is a block diagram of the system interface used in the data storage system of 

10 FIG: 2;- • 

FIG. 6 is a block diagram showing the connections betweert front-end and back-end 
directors to one of a pair of messagenetwork boards used in the system interface of the data 

storage system of FIG. 2; 

FIG. 7 is a block diagram of an exemplary one of the director boards used in the 
15 system interface of he data storage system of FIG. 2; 

FIG. 8 is a block diagram of the system interface used in the data storage system of 

FIG. 2; , : 

FIG. 8A is a diagram of an exemplary global cache memory board used in the system 

interface of FIG. 8; . ;j i . 

20 FIG. 8B is adiagram showmg apair of director boards coupled between a pair of host 

processors and global cache memory feoards used.inthe system interface of FIG. 8; 
' - , FIG. 8C is ablock diagram of mexemp^ 
' ; rear-end directors of me: system mterface of FIG. 8; - . 

•'" • • - FIG! 9 is a^loek:diagram ^ Direct Memory Access (DMA) used in the 

25 ' ' ' '' system interface of the FIG. 8; v . ; , , , ... 

" FIG. 10 is a block di^^ 

FIG. 11 shows the relationship between FIGS, 11A and 11B, suchFIGS. 11A and 

•". l lB together showing aprocess flow diagra^ network 
used in the system interface of nG, 8; . _ : , ... 

FIGS. 1 1C-1 tE : 2e,e^ ^ed by the message network in the 

system interface of FIG. 8 ; ^ : ; ; . > . ; , . _ 
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FIG 1 1 F shows bits in a r mask used- in such message network, 

FIG. 1 1 G shows the result of the mask of FIG. 1 IF applied to the digital word shown 

in FIG. HE; 

FIG 12 shows the relationship between FIGS. 12A and 12B, such FIGS. 12A and 
12B Showing a process flow diagram of the receive operation of a message network used » 

the system interface of FIG. 8; ' ' " a 

FIG 13 shoWs me relationship between nGS. 11A and 11B, such FIGS. llAand 

1 IB together showing a process Cow diagram of the acknowledgement operation of a , t 
message network used in the system interface of FIG. 8; 

FIGS. 14Amdl4B show process flow diagrams of me tr^ 

the transmit DMA of FIG. 9; and ' '--J ■■■:>;■■• • 
, FIGS. ISA and 15B Aow process flow diagrams of me-receiyeDMAoperation of the 

receive DMA of FIG. 10. 

TWTATLED p ireOMPTION 
Referring now to FIG. 2, a data storage system 100 is shown for .referring data 
^een a host computed 120 and abank of disk drives ,40 through a system infcrf^e 
,60 The system interface 160 includes: aplurality of. here 32 front-end duectors 180,-180,, 
coupled to me hos, computer/server 120 via ports-123* a plurality of back-end directs 
MO.^coupledtomebankofdfck^ 

section 240, having a global cache memory 220. coupled to me plurality of ironed 
. airectors 180,180,,: ^^-6ii^M^^^^^^ 

^vemdependenflyof^^^^ 
. Lctors 

' end and'back-end directors 180,-180.. 206.-2G0W u e functionalryisum^r^d mclude a 

• " . CPU controller 314 artd a data pipe 316 to beMescrioed In d«*i. in connect ^n<*5. 
6 and 7. Suffice i, to say here; however, dtat me fon,.,nd andback-end directors 180,-1803, 
200,-200,^1 da*,^ 

les 140 inresponseto messages passing b«ween me dix*iors 180.-.80,,. 200,-200,, 
^ugnmemessagingnelwo*^. The-nessiges faciiitate theda* transfe. ^between b-t_ 
compmer/server .20 and the bank of disk drives .40 W^daupassmg through the global 
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cache memory 220 via the data transfer section 240, More particu.ar.y ; in me case of the 
front-end directors 180,180*. the da* passes between me host computer »*. global cache 
memory 220 through the da* pipe 316 in the Sont-end directors I80,-t80» and me 
messages pass through the message engine/CPU controller 314 in such ftont-end directors 
180,-180* I« the case of the back-end directors 200,200* the data passes between the 
backed directors 200,-200* and me bank of disk drives 140 and rhe global cache memory 
220 through the data pipe 316 in the back-end directors 200,-200* and again the messages 
nasstbrough the message engine/CPU controller 3 14 in such back-end director 200,-2003, 

With such an arrangement, the cache memory 220 in the data transfer section 240 ,s not 
burdened-with the task of transferring the director messaging. Rather the messaging network 
260 operates independent of the dau tranafe section 240 thereby irking the operating 
bindwidfhofthe system interface 160. - - _ 

. i„ aeration, and cor^derhrg tat a r^ 

(ie the host computer/server 120 requests, data from the bank of disk drives 140), the 
request is passed from one of a plurality of, here 32, host computer processors 121,-121,, m 
the host computer 120 to one or more of the pair of the front-end directors 180,-180* 
connected to such host pomputer processor 121,-121,, (It is noted «»tm me host computer 
no each oue of the host computer processors 121, -121* is coupled to here a patr (but not 
Unuted to apair) of the ftont-end directors 180,180*. to provide redundancy in rhe even, of 
a tailure in one of me front end-djrectprs 181,-181,, coupled thereto. Likewise, me bank of 
^ uriv^s !40 has aptaraUty of, : here ? 2,disk drives 141,141,, each diskdnvel4W41« 
■ beingcoupledto here,apair (but not tinted te apair) of the back-end * 
' proWde>redun te cyin.meeven.,of.aMuremoneof^ 

coupled thereto): Each, front-end director 180,180* includes a microprocessor (uP) 299 
(ie- a*entral T rocess ta gurd,XCPtfla^ 

With FIGS. 5 and 7. Suffice itto say here, however, ma. me microprocessor 299 makes a 

« residentcachemanageme* table, not ^ ; ^^^0,^.^>^^ 
access to the resident cache r^gemen. [ table and every time a fo n,-enddu* or 180,180* 

' memory 220 to determine whether «he requested /data" is In the global cache memory 220. If 
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*. guested data is in the global cache memory 220 (U.. a read •'hi."), * front-end director 
,80,-180,,more parucmariy ^microprocessor 299 .herein; mediates a DMA (Dtrect 
Memory Access) operation for the global cache memory 220 and the requested data ,s 
transfenedto the requesting host computer processor 121 , -12l«. 

,f on the other hand, the fronted director 180,-180,, receiving the data request 
determines that the.requested data is no. in the global cache memory 220 (i.e., a W> . . 

CT d direct 180,-180 J2 concludes^ requested dam is in me bank of disk drives .140. 

request for me d- from one of the back-end directors 200,-200,, in order for such back-end 

^^t^*^^*^********- 14 °; The ^°; disk 

wUct back-end direct Z00,-20<*con^ 

drives MOisde^eddtntogapower-upinitializationphase. The map is stored m me 
gI „balcachememory220. Thus, when me ^t-enddirec^r 180,-180,, makes a request for 
Lirom the global cache memory 220 and determines ma. me requested daUts no. m *e 
^oba. cache memory 220 (i.e„ a "miss"), me fronted director l8 0,180„ ts also adv«ed * 
me map in the globaf cache memory 220 of m. backed director 200.-200,, responstbU. for 
me requested data in me bank of disk drives M0. The requesting front-end doctor 180,- 
lW » men must make a request for me. da. in th, of disk drives 140 *«*»>°>» 
d^gnaUdback^nddirector^OO,^. ^^b^«l-<^*^A»- 

Zl inme gill cache memory 200) Uby am*sago*Hchpasse S femmej^ 
.. ^. 2M ,, I.isnoied*en^ 

. : 220 (i.e., does not pass through me^iral^ ^n», r^er^= 8 mrough me 

- memory 220. c~m^:^£^^^^ m ™* M 

used for messaging among me directors 180, W2O0.-200,,. 

Thus on a globafcache memory 220 "read miss", the Som-end d,rec.or 180,-1 80„ 
' '.ends a message «o .he appropriate on. of mVbaCk-ena 0*^200,-200,, through me 
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message network 260 to instruct such back-end director 200 r 20p 32 to transfer the requested # 
data from the bank of disk drives 140 to the glob,al cache memory 220. When accomplished, 
the back-end director. 200 r-200 32 advises the requesting front-end director 180,-180 32 that the 
transfer is accomplished by a message, which. passes. from the back-end director 200i-200 32 
to the front-end director 180,-180 32 through the message. network.260.. In response to the 
acknowledgement signal,. the front-end director, 180,-1 80 32 is mereby.advised that such front- 
end director I80i-180 32 can transfer the data from the global cache .memory 220 to the 
requesting host computer processor 121 r 121 32 as described above when there is a cache 

"read hit". ■ ■ \\ < ; • • v v.! ; ■ • • 

Vi It should be noted that there might be one, or more back-end directors 200i-200 32 
responsible for the requested data. , Thus, if only one back-end director 2001-20052 is 
■ responsible for the requested data, the requesting front-end director 180i-lr80 3 2 sends a uni- 
cast message via the message network 260 to only that specific one of the back-end directors 
200 t -200 32 . On the other hand, if more than one of the back-end directors 200^200^ is 
responsible for the requested data, a multi-cast message (here implemented as a series of uni- 
cast messages) is sent by the requesting one of the front-end directors 180i-180 32 to all of the 
back-end directors 200 u -200 32 having responsibility for the requested data. In any event, 
with both a uni-cast or multi-cast message, such message is passed through the message 
network 260 and not through the data transfer section 240 (i.e.,,not through the global cache 

- memory 220). •. ' ...«-. ..- :,-x-y. =v,:: • 

... I4k0wisc.it shoul4,be*o^^iw^e one. of Ihe host computer processorsmi- 
12t 32 might request data, the acknowledgement signalmay be sent to the requesting iiost 
computer processor 121i;;qr one or more other host computer processors 121,-12132 via a 
multicast (i.e., sequence qf uni-cast) messages through, ttie message network 260 to 
complete the data. read pperationvi < : :i . ; _, . .. 

Considering a.write operation,, fhe host.computer 120 wishes to write data into storage 
(i.e.; into the bank of disk drives, 14,0) :! Dne of uM.frpnt-end directors 180,-180 32 receives the 

., data from the host.computer 420^ writes it u^q.toglobal cache memory 220. The front- 
end director 180i-180 32 then request Ihe.iransfer. pfsuch data after some period of time 
when the back-end director 200, -,200m determines. that the data can be removed from such 
cache memory 220 and stored in the bank, of disk drives 140. Before the transfer to the bank 
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of disk drives 140, the data in the cache memory 220 is tagged with a bit as "fresh data" (i.e., 
data which has not been transferred to fee bank of disk drives 140, that is data which is "write 
pending"). Thus, if there are multiple write requests for the same memory location in the 
global cache memory 220 (e.g., a particular bank account) before being, transferred to the 
bank of disk drives 140, the data is overwritten in the cache memory 220 withthe most 
recent data. Each time data is transferred to the global cache memory. 220, the front-end. 
director I80i-180 32 controlling the transfer^also informs the host computer 120 that the 
transfer is complete to thereby free-up the host computer 120 for other data transfers, . 

When it is time to transfer the data in the global cache memory 220 to the bank of 
disk drives 140, as determined by the back-end director 200 l -200 32 , the back-end director 
200r200 32 transfers the data from the global cache memory 220 to the bank of disk drives 
140 and resets the tag.associated with data in the global cache-memory 220 <Le., un-tags the 
data) to indicate that the data in the global cache memory 220 has been transferred to the 
bank of disk drives 140! It is noted that the un-tagged data in the global cache memory 220 
remains there until overwritten with new data. 

Referring now to FIGS. 3 and 4, the system interface 160 is shown to include an 
electrical cabinet 306having stored therein: a plurality of, here eight front-end director 
boards 190i-1908, each one having here four of the front-end directors 180 s -180 32 ; a plurality 
of, here eight back-end director boards 210r210 8 , each one having here four of the back-end 
directors 200,-200 32 ; and a plurality of, here eight, memory boards 220' which together make 
up the-global cabhe memory 220. These boaros^iug into the front side of a backplane 302. 
^t is noted that the backplane 302 is 'i ; rmd-plane rpimied circuit board). Plugged into' the 
backside of m^ 

backplane'302 has plugged into it adapter boards- not shown in FIGS .2-A, which couple the 
boards plugged into the back-side of the backplane 302'with the computer 120 and the bank 
of disk drives 140 as shown in FIG. 2. that is/relernrig again briefly to FIG. 2, an I/O 
adapter, not shown, is coupled between each one of the front-end directors 180 r l80 32 and the 
host computer 120 and an VO adapted, hot shbwn/is coupled between each one of the back- 
end directors 200i-200 32 and m e b ank of disk dnves t'40; 

Referring now toFIG. 5, me system internee 1604s shown to include the director 
boards 190,.190 8 , 210,-2io 8 and the global cache memory 220. plugged into the backplane 
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302 and the disk drives 141 r 141 32 .in the bank, of disk drives along with the host computer , 
120 also plugged into the .backp.lane.302 via I/O adapter boards, not shown; The message 
network 260 (FIG, 2) includes the message, network boards 304, and 304 2 . Each one of the 
message network boards 304, and 304 2 is identical in construction. A pair of message 
network boards, 304, and 304 2 is used for redundancy and for message load balancing. Thus, 
each message network board 304,, 304 2 , includes a controller 306, (i.e., an initialization and 
diagnostic processor comprising a CPU,.system controller interface and memory, as shown in 
: FIG. 6 for one of the message network boards 304, , 3042, here board 304.) and a crossbar 
switch section 308 (e.g., a switching fabric made up of here four switches 308,-3084)- 

Referring again to FIG. 5, each one of the director boards 190,-210 8 includes, as 
noted above four of the directors 180,.180 32 , 20 0l -200 32 (FIG. 2). It is noted that the director, 
boards 190,-190 8 having four front-end directors per board, 180,.l80 32 -are«ferredto as 
front-end directors, and the director boards 210,-210 8 having four back-end directors per 
board, 200,-20032 are referred to as back-end directors. Each one of the directors 180,-18032, 
200,-200 32 includes a CPU 3 10, a RAM 3 12 (which make up the microprocessor 299 
referred to above), the message engine/CPU controller 314, and the datapipe 316. 

Eachbneofme.directorboardsl90 1 -210 8 includesacrossbarswitch318. The 

. crossbar switch 318 has four input/output ports 3 19, each one being coupled to the data pipe 
316 of a corresponding one.of the four, directors 180,-180*. 200,-200 32 on the director 
board 190,-210 8 . The crossbar switch 3.18 has eight output/input ports collectively identified 
■ in FIG 5 by numerical designation Thecrossbar 

* switch 318 on the fronted director boards 191, r 191 8 is used for coupling the data pipe 316 
ofa selected one of me fo 

- 190,-190^0 the global cache memory.220 via the backplane 302 and I/O adapter, not shown. 
The crossbar switch 318 on the back-end director boards 210,-210, is used for coupling the 
data pipe V6 of a selected one of the four backed directors 200,-2003 2 on the back-end 
director board 210 : -210 8 to, the global cache memory 220 via the backplane 302 and I/O 
- adapter, not shown, preferring to FIG. 2, the data pipe 3 16 in the front-end directors 
180,-180 32 cou P les data between the host computer 120 and the global cache memory 220 
while the data pipe 316 in the back-end directors 200,-200* couples data between the bank 
Of disk drives 140 and the global cache memory.220. 'ii iVrioted that there are separate point- 
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to-point data paths P.-P* (FIG. 2) between each one of the directors 180 r 130 32 , 200,-200 32 
and the global cache memory 220. It i£ also noted that the backplane 302 is a passive 
backplane because it is made up of only etched conductors on one or more layers of aprinted 
circuit board. That is, the backplane 302 does not have any active components. 

Referring again, to FIG. 5, each one of the director boards 190 ,-2108- includes a 
crossbar switch 320. Each crossbar switch 320 has four input/output ports 323, each one of 
the four input/output ports 323 being coupled to the message engine/CPU controller 314 of a 
corresponding one of the four directors' lSOrlSO*, 200,-20032 on the director board'190,- 
210 8 . Each crossbar switch 320 has a pair of output/input ports 325,, 325 2 , which plug into 
the backplane 302. Each port 325, -325 2 is coupled to a corresponding one of the message 
network boards 304,, 304., respectively, through the backplane 302. The crossbar switch 
320 on the front-end director boards 190,-190 8 is used to couple the messages between the 
message engine/CPUcontroUer 314 of a selected one of the four front-end directors 180,- 
1 80 32 on the front-end director boards 190,-190 8 and the message network 260, FIG. 2. 
Likewise, the back-end director boards 210,-210 8 are used to couple the messages produced 
by a selected one of the four back-end directors 200,-200 32 on the back-end director board 
210,-210 8 between the message engine/CPU controller 314 of a selected one of such four 
back-end directors and the message network 260 (FIG. 2). Thus, referring also to FIG. 2, 
instead of having a separate dedicated- message path between each oneof the directors 180,- 
180 32 200,-20032 and the message network 260 bvmch womd require M individual <> 
connitipr^ to me backplane 302 for each of ^^^^^ 

'm/4 individual connections are required}. Thus^ the total number of connections between the 
-directors^ I**.- T*us, it should be 

, noted from FIGS. 2 and 5 that ^e messagb network 260 (FIG:*) includes the crossbar switch 
320 and the message network boards 304,, i64 2 . 

: •,. ' Each message is' a o^byte' descriptor/^nl F*G: 2Ay Which is created by the 
CPU 310. (FIG. 5) under software controt and is stored ma send queue in RAM 312. When 
the message is to be read from the send queue in RAM 312 and transmitted through the 
message network 260.(FIG: 2) to one or more other directors via a DMA operation to be 

■ described, it is packetized in ie packetizer por^ (FIG. 7) 

into a MAC type packet, shown in FIG. 2B, here using the NGIO protocol specification. 
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There are three types of packets: a message packet section; an acknowledgement packet; and . 
a message network fabric management packet, the latter, being used .0 establish the message 
network routing during initialization (i.e., during power-up).. Each one of the MAC packets 
has . an 8-bvte header which includes source (Le.. transmintog director) and destination (,e.. 
receiving director) address; a payload; and terminates. with a4-by«e CycUc 
Check (CRC) as shown in FIG. 2B. The acknowledgement packet (i.e., signal) has a 4-byte 
■ acknowledg^entpayload section.. The message Packet has a 32-by.e payload secnon. The 
Fabric Management Packet (FMP) has a^epayload section^ MAC packet « sen. 
to *. crossbar switch 320. The destination. portion of theoacke. is used toindicate the 
destination for thotnessage and isdecodedby the switch 320 to determine which port ti>e 
message is to be routed. The decoding process us« a decoder tab.e 327 in me switch 
suchtoble being initiahzedby contouring power-up by the irdtializa^oa.and jhagnostic 
processor (controller) 306 (FIG. 5). The table 327 (FIG. 7) provides the relationsmp between 
destination address portionof the MAC packet, which identifies tire routing for the 

- * i an,, 200, -200^ on me director board I90r 

message and the one of the four directors I80 r 180 32 , 2Wi /w 32 on 

^r^rV hoards 304i 3042 to which the message is 
190 8 , 21Qi-210 8 or to one of the message network boards 3Wi, jwi 

to: be directed. . ,. „ ^ *>« v>< i« 

More particularly, and referring to FIG. 5, a pair of output/mput ports 325,, 325 2 ,s 

provided for each on. of the crossbar synches 320, each one being coupled to a 
Responding one of me pair of message network.boards 304,, 304, Tta. each one of*. 

counledtoacorres^ 

. corresponding one ofthe.directorboards } 90,^,n<kf™> *? b • ^ 

. ^alsoh^esthreec^ 

» tt «^*^.^^S^ FI ?-. < - ^. cons^enng message 
networkboard 304,, mput^umu, ports 322,^22, are coupled to ou«put/u,pu, ports 325, 
. .. front-enddh^tor*.^^^ 

^utput/inpu, p<,r^ , led 

* ^ io< rt f frnnt-end director boards 190r 190 8 ana 
via the backplane 302, to output/input potts 325 2 of front end airec 
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input/output ports 322 9 -322 !6 are coupled; via the backplane 3C2, to output/input ports 325 2 
of back-end director boards 210i-2i08.' 

As noted above, each one of the message network boards 304,; 304 2 includes a 
processor 306 (FIG. 5) and a crossbar switch section 308 having four switches 308,-308 4 , as 
shown in FIGS. 5 aha 6. The switches 308,-308 4 are interconnected as shown so that 
messages can pass between any pair of the input/output ports 322 1 -322 16 . Thus, it foilow 
that a message from any one of the front-end directors 180 r 180 32 can be coupled to another 
one of the front-end directors 186 i .i 80 3 2 and/or .to any one of the back-end directors 200 1- 
200 32 . Likewise, a message from any one of the back-end directors 180,-180 32 can be 
coupled to another one of the back-end direct6rs'180,.180 3 2 and/or .to any one of the front- 
end directors 200 l -200 32 . : ... 

, As noted above, each MAC packet (FIG. 2B) includes in an addf ess destination 
portion and a data payload portion. The MAC header is used to indicate the destination for 
the MAC packet and such MAC header is decoded by the switch to determine which port the 
MAC packet is to be routed. The decoding process uses a table in the switch 308,-3O8 4 , such 
table being initialized by processor 306 during power-up. The table provides the relationship 
between the MAC header, which identifies the destination for the MAC packet and the route 
to be taken through the message network. Thus, after initialization, the switches 320 and the 
switches 308,-308 4 in switch section 308 provides packet routing which enables each one of 
the directors 180i-180 32 , 200i-2003 2 to transmit a "message betweenitself and any other, one 
of the directors, regardless of whemer such ouier^ector is on me same director board 190,- 
1908, 21oi--210 8 or on a different director board. ' Furmer, the MAC packet has an additional 
bit Bin the header thereof, as shown mFia through 
message network board 304, or through message network board 304 2 . Dming normal" 
operation, this additional bit B is toggled between^ldgic f and a logic 0'so that one message 
passes through one of the redundant message networkboards 304,, 3042 and the next 
message to pass through the other one of the message network boards 304 u 3042 to balance 
the load requirement on me system^ How^ of a failure in one of the message 

network boards 304,, 304 2 , the non-failedone of the boards 304,, 3042 is used exclusively 
until the failed message network board is replaced. 
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Referring now to FIG. 7, an exemplary one of the director boards 190 ,-190,, 210,- . 
210 8 here director boar* 190, is shown to include directors 180,, 180 3 , 180, and 180 7 . An 
exemplary one ofthe directors 180,-1 80 4 , here director 180, is shown in detail to include the 
data pipe 316, the message engine/CPU controller 3.14, the RAM 3 12, and the CPU 310 all 
coupled to the CPU interface bus 317, as shown. The exemplary director 180, also includes: 
a local cache memory 319 (which is coupled to the CPU 310); the crossbar switch 318; and, 
the crossbar switch 320, described briefly above in connection with FIGS. 5 and 6. The data 
pipe.316 includes a protocol translator 400, a quad port RAM. 402 and a quad port RAM 
controller 404- arranged as shown. Briefly.the protocol translator 400 converts between the 
protocol ofthe host computer 120, in the case of a front-end director 180^180^ (and 
between the protocol used by the disk drives in .bank 140 in the case of a back-end director 
200,-2003.) and the protocol between the directors 180,-1803, 200,-200^ andihe global 
memory 220 (FIG. 2), More particularly, the protocol used the host computer 120 may, for 
example, be fibre channel, SCSI, ESCON or FICON, for example, as detemuned by the 
m anufacture of the host computer 120 while the protocol used internal to the system interface 
160 (FIG 2) may be selected by the manufacturer ofthe interface 160. The quad port RAM 
402 is a FIFO controlled by controller 404 because the rate data coming into the RAM 402 
may -be different from the rate data leaving the RAM 402. The RAM 402 has four ports, 
each adapted to handle an 18 bit digital- word. Here, the protocol translator 400 produces 36 
bit digital words foKthe system interface 160 (FIG. 2) protocol, one 18 bit portion ofthe 
word is coupled to one of a pair ofthe ports of the quad port RA^ 402 and the other 18 bit 

■ portion ofmeword is coupl*^ 
402 The quad port . RAM has ^ 

■ beingadaptedtohand^ 

independently con^^^ 
: withi n the RAM 402, DaUis ^^bet^cen the ports 402A, 402B and the cache 

memory 220 (FIG. 2) through the crossbar, switch 318, as shown. 
• ... -The crossbar swit^ Eachoneofthe 
switches 406A, 406B includes fpur input/output director-side ports D,-D 4 (collectively 
referred to above in connection with FIG, 5 as port 319) and four input/output memory-side 
ports M.-M,, M 5 -M„ respectively,' as indicated. The input/output memory-side ports M.-M,, 
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M 5 -M 8 were collectively referred to ^^abo^ViH connection with FIG. 5 as port 317). The 
director-side ports Di-D 4 of switch 406X ; are connected to the 402A ports of the quad port 
RAMs.402 in each one the directors 180,, I8O3, 18O5 and 180 7 , as indicated. Likewise, 
director-side, ports of switch 406B are'cofmected to the 402B ports of the quad port RAMs 
402 in each one the directors 180,, ISO,, 180 5 , and 1807. as indicated: The ports D,-D 4 are 
selectively coupled to the ports Mi-& in'^cdrdance with control words provided- to the 
switch 406A by the controllers mdifectors i'80,, 180 3 , 180 5 , 18O7 on busses R a ,-Ra* 
respectively, and the ports D1-D4 are copied Sports Ms-Ma in accordance with decontrol 
words provided to switch 406B by the conrr6Uers"in directors 180,, 180 3 , 180 5i 180 7 on 
busses RBI-R84, as indicated. The sigrials on buses^RArRM are request signals. Thus, port 
402A of any one of the directors 180», ISO,', lSOsVl^rhay be coupled to any one of the 
ports M1-M4 of switcrr406A, selectively-in accordance with the request-signals on buses 
R M -R A4 . Likewise, port 402B of any one of the directors 180,-180 4 may be coupled to any 
one of the ports M 5 -M 8 of switch 406B, selectively in accordance with the request signals on 
buses Rb!-Rb4. The coupling between the Sector boards 190,-1908, 210,-210 8 and the 
global cache memory 220 is shown in FIG. 8. 

More particularly, and referring also to FIG. 2, as noted above, each one of the host 
computer processors 121, -I2I32 in the host computer 120 is coupled to a pair-of the front- 
end directors 180,-180 32 . to provide redundancy in'me event of a failure in one of the front 
end-directors 18U-181* coupled thereto? LikewisVtfae bank of disk drives 140 has a* 
plurality of, here 32, disk' drives i 41 ^i^^eM^sk drive t4lW41 32 being coupled to a 
* ^ of the backed dkecto'ri 200,-200*. ^^^km^miho^o^ failure in 
one of the back-end directors ^ibb* cb^ exemplary host 

computer processorl21,,suc^c^ 
180 2 Thus, if director 180, fai^ the host ^ 

system interfacel60, albeit by theoth«^ ' ^ 18 ° l ttd 

, 180 2 are considered redundancy pairs' of directorsrLikewise,bmer redundancy pairs of 
front-end directors are: front-end d^torshsCt^i f 80s, 180 6; 180 7 , 180 8: 180,. 180, 0 ; 
180,, 180„; I80u, 180V 180, s . 180, 6; l80 l7 , i^ 8 : ! lfei80 20 ; 180 2 „ 180 22; 180 23 , 180 24; 
l 180 25 , 180*; 180 27 , 180 28 ?180 2 ;: 18b 30 ; and l80 » befa * 

shown in FIG. 2). 
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• • Likewise, disk drive 141 , is coupled to a pair of back-end directors 200,, 200 2 . Thus, - 
if director 200, fails.the disk drive 141, can still access the system interface 160, albeit by the 
other back-end director 180 2 , Thus, directors 200, and 200 2 are considered redundancy pairs 
of directors. Likewise, other redundancy pairs of back-end directors are: back-end directors 
200 3 , 200 4 ; 200s, .200,6; 2OO7. 200 8; 2OO9. 2pp,o; 2Q0, ,, 200,2; 200,3, 200, 4; 200, 5 , 200 1 6; 200,7, 
200, 8 ; 200,9, 200 20 ; 200 21 , 200 22 ;200 23 . 200 24; 200 25; , 200 26 ; 200 27 , 200 28 ; 200 29 , 200 30 ; and 
■:200 3 i, 200 32 (only directors 200 3 , and 200 32 being shown in FIG. 2). Further, referring also to 
FIG. 8, the global cache memory 220 includes a plurality of, here eight, cache memory 
boards 220,-220 8 , as shown. Still further, referring to FIG. 8 A, an exemplary one of the 
cache memory boards, here board 220, is shown in detail and is described in detail in U. S. 
PatentNo. 5,943,287 entitled ;'Fault,Tolerant Memory System", John K. Walton, inventor, 
issued August 24, 1999 and assigned to the same assignee as the present invention, the entire 
subject matter therein being incorporated herein by reference. Thus, as shown in FIG. 8A, 
the board 220, includes a plurality of, here four RAM memory arrays, each one of the arrays 
has a pair of redundant ports, i.e., an A port and a B port. The board itself has sixteen ports; 
a set of eight A ports M A ,-M A8 and a set of eight B ports M B i-M B8 . Four of the eight A port, 
here A ports M a ,-Mm are coupled to the M, port of each of the front-end director boards 
190,, 190 3 , 190 5 , and 190?, respectively, as indicated in FIG. 8. Four of the eight B port, here 
B ports M B i-Mb4 are coupled to the M, port of each of the front-end director boards 190 2 , 
190 4 , 190 6 , and 190 8 , respectively, as indicated in FIG. 8 . The other four of the eight A port, 
here A. ports Ma5-M AS are coupledto isMiport of eachof the back-end director boards 
,210,, 210 3 , 2,10;, and 2IO7, respectiyMy, ; as mciicated in FIG. 8. The other fouMf the eight B 
port, here S ports Mbs-Mw are coupled to the M, port of each of the back-end director boards 
210 2 , 2104, ^.106, and 219s, respectively, as mdicated in FIG. 8 

Considering the exemplary ,four f A, ports M.A r M A 4, each one of the four A ports M AI - 
M A4 can be coupled to the A port of^any one of the memory "arrays through the logic network 
221 l A . Thus, considering port M A , , such port can be coupled to the A port of the four 
memory arrays. Likewise, considering me, four A ports M A5 -M A8 , each one of the four A 
.ports Mas-Mas can be coupled to the A port of any one of the memory arrays through the 
logic network 22 1, B . Likewise, considering the four B ports Mbi-Mem, each one of the four 
B ports M B i-Mb4 can be coupled to the B .port of any one of the memory arrays through logic 
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network 221 m.' Likewise, considering the four B ports Mbj-Mm, each one of the four B ports 
MbtMm can be coupled to the B port of anyone of the meniory arrays through the logic 
network 221 2B . Thus, considering port Mat. such port can be coupled to the B port of the 
four memory arrays. Thus, there are two v paths data and control from either a front-end 
director 1 80,-1 80 32 or a back-end director 200, -200 32 can reach each one of the four memory 
arrays on the memory board. Thus, there are eight sets of redundant ports on a memory 
board, i.e., ports M A „ M B1 ; M^, M B2 ; M^, M B3 ; M A4 , Mb4; M^, M bs ; M Afi , M B6 ; M A7 , M B7 ; 
and M A8 , Mb* Further, as noted above eachfone of the directors has a pair of redundant 
ports, i.e. a 402A port and a 402 B port (FIG; 7). Thus, for each pair of redundant directors, 
the A port (i.e., port 402A) of one of me difebtors in the pair is connected to one of the pair 
of redundant memory ports and the B port (i.e, 402B) of the other one of the directors in 
such pair is connected-to the other one of thepair of redundant memory ports... 

More particularly, referring to FIG. 8B, an exemplary pair of redundant directors is 
shown, here, for example, front-end director 180, and front end-director 180,. Itisfirst 
noted that the directors 180,, 180, in each redundant pair of directors must be on different 
director boards, here boards 190,, 190., respectively. Thus, here front-end director boards 
190,-1908 have thereon: front-end directors 180,. 180 3 , 180 5 and ISO,; front-end directors 
180 2 180 4 , 180 6 and 180 8 ; front end directors 180 9 . 180„, i80, 3 and 180, 5 ; front end 
directors 180, 0 . 180, 2 , 180 14 and 180 16 ; front-end directors 180, 7 . 180, 9 , 180., and 180.3; 
front-end directors 180, 8 . 180,0, 180.. and 180.4; front-end directors 180 25 . 180.7, 18^ and 
. 180 3 , • fronted directors' 180, 8 :i80 2 o, 180^ I80ii: thus; here back-end directorial 
' 21Q,-210 8 have thereon: back-end' directors 20di! 200 3 , 200 5 ah ! d 200 7 vback-end^eetbrs 
" 2 oo 2 2004, 200 6 J and' 200 8 ; back-end directors 20b 9 ; 200,,/206, 3 arid 200, 5 ; back-end • 

directors 200, 0 . 200, 2 , 200, 4 and 200, 6 ; back-end'd^ctors^OOW. 200, 9 , 200., and 200 23 ; 
, back-end directors 200, 8 . 200 20 , 200,, and 2^^^ectors 200 25 ;200 27 , 200. 9 and 
200 3I - back-end directors 200, 8 , 200.0, 200^ U ana 2b6.4; 

.' ' Thus, here front-end director 180, Jshown in FIG. 8A, is oh front-end director board 
190, and its redundant front-end director iS^hown in FIG. 8B, is on anther front-end 
director board, here for example: fr6nt-end^irect6r board 190,. As described above, the port 
.' 4 o2 A of the quad port RAM 402 (Le., the A port referred to above) is connected to switch 
406A of crossbar switch 318 and the port 402B of the quad port RAM 402 (i.e., "the B port 
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referred to above) is connected ;to switch 406B of crossbar switch 3 18. Likewise, for 
redundant director 180 2 , However, the ports M1-M4 of switch 406A of director 180 , are 
connected to the M A i ports of global cache memory boards 220,-200 4 , as shown, while for its 
redundancy director I8O2, the ports M,-M4 of switch 406 A are connected to the redundant 
Mbi ports ofglobal cache memory boards 220i-2004, as shown. . . • ■ 

Referring in more detail to the crossbar switch 3 1 8 (FIG. 7), as noted above, each one 
of the director boards 190,-210 8 has such a switch 318 and such switch.3 18 includes a pair of 
switches 406A, 406B. Each one of the switches 406A, 406B is identical in construction, an 
exemplary one thereof, here switch 406A being shown in detail in FIG. 8C. Thus switch 
406A includes four mput/outputdirector-side ports D, : D 4 as described in connection with 
exemplary director board 190i. Thus, for me director board 190i shown in FIG. 7, the four 
inputfoutput director-side. ports D,-D 4 of switch 406A are eachcoupled toTthe-port 402A of a 
corresponding one of the directors 180,, I8O3, 18O5, and I8O7 on.the director board 190 t . 

Referring again to FIG. 8C, the exemplary switch 406A includes a plurality of, here 
four, switch sections 430i-430 4 . Each one of the switch sections 430,-430 4 is identical in 
construction and is coupled between a corresponding one of the input/output director-side 
ports D1-D4 and a corresponding one of the output/input memory-side ports Mi-M^ 
respectively, as shown, at should be understood that the output/input memory-side ports of 
switch 406B (FIG. 7) are designated as, ports M 5 -M 8 , as shown. It should also be understood 
that while switch 406A is responsive tat request .signals on busses Ra!-Ra4 from quad port 
controller 404 in directors 180,,, lZ(h«Sf&s*<l*fh C^G. 7), swUch 406B is responsive in like 
manner to request signals on busses.-RBi-RB4 from controller, 404 in directors 180,, I8O3, 1 80s 
an4 18Q 7 ). Morefparticularly, controller 404 of director 130., produces, request signals on 
busses Rah or R B ,. ^ maike^manner^qntroller 40^ pf director I8O3 produces request signals 
on busses R/u or Rb* controller 4§£ pf director 180 5 produces request signals on busses Raj . 
or Rbj, andsontrqller 404 oCd^tOf ! .1.8%.fp4uce5 ; requ.est signals on busses R A4 or R*. 

Considering exemplary switch:secti0n;.43Qu such switch section 403, is shown in 
FIG. 8C to include a FIFO 432 fed^y„the request signal pn bus R, A . (It should be .. . 
understood that the FIFOSj not shown, in, switch sections 430 2 -430 4 are fed by request signals 
R A 2-Ra4, respectively).^ The switch.5ecti.0n 4.06, also includes.a request generation 434, and 
, arbiter 436,.and selectors 442 and.446,.aU arrangedas shown. The data at the memory-side 
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ports M1-M4 are on busses DMl^DM4>ar£fedas inputs to selector 446. Also fed to selector 
446 is a control signal produced by the'tequest generator on bus 449 in response to the 
request signal R A i stored iri FIFO 432. The control signal on bus 449. indicates to the selector 
446 the one of the memory-side ports Mi-M* which is to be coupledto director-side port D|. 
The other switch sections 43 0 2 -430 4 operate in like manner with regard to director-side ports 
Di-D 4 ; respectively and me memory-side ports M1-M4. 

It isto be noted that the data portion of the word at port Di (i.e., the word on bus 
DDI) is also coupled to the other switch 'sections '4302-430 4 . It is further noted thatthe data 
portion of the words at ports D 2 -D 4 ' (i.e M "the words on busses DD2-DD4, respectively), are 
fed to the switch sections 430i-430 4 , abdicated. That is, each one of the switch sections 
430i-4304 has the data portion of the words on ports Dt-D 4 (i.e., busses DDI -DD4), as 
indicated. It is also noted -that the data portion of the word at port Mv (i-<Cthe word on bus 
DM1) is also coupled to the other switch sections 430 2 -430 4 ." It if further noted that the data 
portion of the words at ports M2-M4 (Le.'," the words on busses DM2-DM4, respectively), are 
fed to the switch sections 4302-430 4 , as indicated. That is, each one of the switch sections 
430i-430 4 has the data portion of the words on ports Mi -Mi (i.e., busses DM1 -DM4), as 
indicated. 

As will be described in more detail below, a request on bus Rai to switch section 430, 
is a request from the director 1 8O1 which identifies the one of the four ports M t -M4 in switch 
430i is to be coupled to port 402A of director 1 Sty (director side port Di). Thus, port 402A 
of director I8O1 rriay be coupled to 'one of the 1 memory side pofts'Mt-Mi selectively in 
' accordance with me d^ta bn bus r RA». likewise, a Request oh%uses R^2f Ras, R/w to switch 
section 430r430 4 , respectively, are requests from the directdis-l 80 3 ,- ISQ5, and I8O7, - 
respectively, which identifies theohe of the four ports Mr"M 4 m ; switch-430r-430 4 is to be • 
coupled to port 402A of directors I8O3, 180is T and c l ^.respectively;- 

More particularly, the requests Rai are stored as they are produced by the quad port 
RAM controller 440 (FIG. 7) in receive FIFO '432. The request generator 434 receives from 
FIFO 432 the requests' and determines'which due of the four memory-side ports M-.-M4 is to 
be coupled to port 402 Ajof director I 8O1 : " These requests for. memory-side ports M1-M4 are 
produced on lines RAl.l - RA1,4, fespectivei y r thus, line RAl.l (i.e., the request for 
memory side port M,) is fed to arbiter 436 arid the ^quests from switch sections 430 2 -430 4 
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(which are coupled to port 402A of directors ^ and 180,) on line RA2.1. RA3.1 and . 
RA41 resp«ctivelyarealsofed.o.thearb i .er436,.asindica.ed. . Tbe-bto43«««^ 
multiple requests for memory-side port M, on a.first come-first serve basis. The arbiter « 6 
thenproduces acontrol signal on bus 435 indicating the one of the directors 180,, 180,, ISO, 
or I8O7 which is to be coupled to memory-side port Mi . 

The control signal on bus 435 is fed.to selector 442., Also fed to selector 442 are the 
dataportionof the data at port Di, i.e ; , me data on data bus DDI) alon^ with the data portion 
of me dat*at ports DrD* i.e.. the data on data busses.pD2-.pD4, respectively, as indicated. 
Thus me control signal on bus 435 -causes *e selector 442 to couple |o the output thereof the 
o ^bussesDDl-DD4frommeoneof,mednec«»rs.l80 1 , 180,, 180 5 .180, being granted 
■aocess t omemory-si^po rt M,by, ft earbiter436, The selected outputs of selector 442 is 
coupled to memory-side pert Mi. It should be noted that when the arbiter436 receives a 
• .,. _,.i.™li^RAl 1 RA21 RA3,l and RA4.1, acknowledgements are 
request via the signals on lines tuy* >f> x^rw, 

returned by the arbiter 436 via acknowledgement signals on line AK1.1, AK1 2, AK1.3, 
AK1.4, respectively such signals being fed to the request generators 434 to switch sectton 
4301,4302,4303,4304, respectively. 

Thus, the dataon any port D,-D 4 canbe coupled to and one of the ports M.-M. to 
effectuate the point-to-point data paths Pl -P« described above in connection wrth FIG. 2. 

Referring again to FIG. 7, da^from host computer 120 (FIG. 2) is presented to the 
: system interface 160 (FIG. 2) in batches from many host computer processors I21i-121 3 2^ 
., Thus W*ta.*»*W^i* i ? l,:i2fa«.tt«^»«*«^«te 
as they are presented u, a director i^lSoi The batch from each Host computer processor 
. 80, ISO, (i.e...sourc,) is tagged by me protocol translator 400. More particularly by a 
Tacheou ASIC in the case of a fibre channe. connection. The controller 404 has a look-up 
: table formed during initialization.^ *c data comes into me protocol translator 400 and .s 

■ , 400 informs me controller that the data is, in the quad port RAM 420." The controller 404 
, .ooksatme-corJigurationoW 

. ■*^^^«^wm-w^«?'*"^*"'- Thecontfoller 

404 thus produces the request signals on me' appropriatebus.R.i.Re^andmentellsmeo^d 
port RAM 402 that there is a b.ock of data a, a particular location in the quad port RAM 402, 
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move it to the particular location in the glbbal'cache memory 220. The crossbar switch 3 1 8 
also takes a look at what other controllers 404 in the directors 180 3 , 130 5 , and I8O7 on that 
particular director board 190, are asking by making request signal on busses Ra2, Rbz, Ra3, 
Rb3, R A 4, Rb4, respectively. The arbitration of multiple requests is handled by the arbiter 436 
as described above in connection with FIG. 8C. 

Referring, again to FIG. 7, the exerhplary director 180, is shown to include in the 
message engine/CPU controller 3 14. the message engine/CPU controller 314 is contained in 
a field programmable gate array (FPGA). Tne message engine (ME) 315 is coupled to the 
CPU bus 317 and the DMA section 408 as shown. The message engine (ME) 315 includes a 
Direct Memory Access (DMA) section 408, a message engine (ME) state machine 410, a 
transmit buffer 424 and receive buffer 424, a MAC packetizer/depacketizer 428, send and 
receive pointer registers 420, and a parity generator 321. The DMA section 408 includes a 
DMA transmitter 418, shown and to be described below in detail in connection with FIG. 9, 
and a DMA receiver 424, shown and to be described below in detail in connection with 
FIG.10, each of which is coupled to the CPU bus interface 317, as shown in FIG. 7. The 
message engine (ME) 315 includes a transmit data buffer 422 coupled to the DMA 
transmitter 4 1 8, a receive data buffer 424 coupled to the DMA receiver 42 1 , registers 420 
coupled to the CPU bus 317 through an address decoder 401, the packetizer/de-packetizer 
428, described above, coupled to the transmit data buffer 422, the receive data buffer 424 and 
the crossbar switch 320, as shown, and a pariry generator 321 coupled between the transmit 
data buffer 422 and the cr^b» swiUi>26: More partitalatly. the packetizerportion 428P 
is used to packetize the message payfead into bMc packe t (FIG. 2B) passing-iron* the 
. transmit data buffer 422 to the crossbar switch 320 and the de-packe'tizer portion 428D is 
, used to de-packetize the MAC packet into message pay load data passing ficm the crossbar 
...switch 320 to the receive date buf^i Thepacketiza^onis here performed by -a MAC 
core which builds a MAC packet and appends to i &cn message ; such tmngs.as a source and 
destination address designation ^ indicating the director sending and receiving the message and 
a cyclic redundancy check (CRC), as ^ described above. The message engine (ME) 315 also 
includes: a receive write pointer 450, a receive read pointer 452; a send write pointer 454, 
and a send read pointer 456. 
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Referring now to FIGS. I i and 12, the transmission of a message from a director 
180i- 1 80 32 , 2OO1-2OO32 and the reception of a message by a director 2 10,-2 10 32 , here 
exemplary director 180, shown in FIG. 7) will be described: Considering first transmission 
of a message, reference is made to FIGS. 7 and 1 1. First, as noted above, at power-up the 
controller 306 (FIG- 5) of both message network boards 304,, 304 2 initialize the message 
routing mapping described above for the switches 308 ,-3 08 4 in switch section 308 "and for the 
crossbar switches 320. As noted above, a request is made by the host computer 120. the 
request is sent to the protocol translator 400. /The protocpl.translator 400 sends the request to 
the microprocessor 299 via CPU bus 3 17 and buffer 301 : When the CPU 3 1 0 (FIG. 7) in the 
microprocessor 299 of exemplary director 180, determines that a message is to be sent to 
/another, one of the directors 1802rl80 32 , 20Qr200 3 2, (e.g.,.the CPU 310 determines that there 
has been a "miss" in the global cache memory 220 (FIG. 2) and wants to s-end_a message to 
the appropriate one of the back-end directors 200,-200 32 , as described above in connection 
with FIG. 2), the CPU 310 builds a 64 byte descriptor (FIG. 2A) which includes a 32 byte 
message payload indicating the addresses of the batch of data to be read from the bank of 
disk drives 140 (FIG. 2) (Step 500) and a 32 byte command field (Step 510) which indicates 
the message destination, via an 8-byte bit vector, i.e., the director, or directors, which are to 
receive the message. An 8-byte portion of the command field indicates the director or 
directors, which are to receive the message. That is, each one of the 64 bits in the 8-byte 
; portion corresponds to one :of the 64 d^tors... Here, a logic 1 in a bit indicates thatthe 
corresponding director is to receive a r^essage^and a logic 0 indicates that such corresponding 
, director is not to receive #e message... Thus," if the 8-byte word .has i more than one logic 1 bit 
more than one director will receive the same message. As will be described, the same 
message will^ 

. sent sequentially to all such direc^rs, t ta any event, the resulting 64-byte descriptor is 
■ generated by the CPU3J0 .(FIG J}^? 512) is written into the RAM 312 (Step 514), as 

shown in FIG. 11. • - .. . • v .. t 

More parucuhriy, the^R^' 5liirp^&faofq^cs; aiend queue and -a 'receive 
queue, as shown in FIG, jj r The RAM ^12 is coupled to the CPU bus 317 through an Error 
Detection and Correction (EDAC)/Memory control section 303, as shown. The CPU 310 
. ; then indicates to the message engine : (ME) 31 5 stote machine 410 (FIG. 7) that a descriptor 
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has been written into the RAM ill' irshould be noted thatthe message engine (ME) 3 15 
also includes: a receive write pointer blunter 450, : the received pointer or counter 452, 
the send write pointer or counter 454, aid the send read pointer or counter 454, shown in 
FIG. 7. All four pointers 450, 452, 454 : and~456 are reset to zero ohpower-up. Asisalso 
noted above, the message engine/CPU controller 3 14 also includes: me-de^paeketizer portion 
428D of packetizer/de-packetizer 428, coupled to the receive data buffer 424 (FIG. .7) and a 
packetizer portion 428P of the packeW<te-^ketizer 428, coupled to the transmit data, 
buffer 422 (FIG. 7). thus, referring agam^to ttG. 11, when the CPU 310 indicates that a 
descriptor has been written into the RAM312 and is now ready to be sent, the CPU 310. 
increments the send write pointer and sendJ to thesend write pointer register 454 via the 
register decoder 401. Thus, the Contents of the send write pointer register 454 indicates the 
number of messages in the send queue il2S of RAM 3 12, which haveuo^n sent. The 
state machine 410 checks the send write pointer register 454 and the send read pointer 
register 456, Step 5 1 8. As noted above, both the send write pointer register 454 and the send 
read pointer register 456 are initially reset to zero during power-up. Thus, if the send read 
pointer register 456 and the send write pointer register 454 are different, the state machine 
knows that there is a message is in RAM 312 and that such message is ready for 
transmission. If a message is to be sent, the state machine 410 initiates a transfer of the. 
stored 64-byte descriptor to the message engine (ME) 315 via the DMA transmitter 418, FIG. 
7 (Steps 520, 522). The descriptor is sent fronT t& send queues 312S in RAM 312 untilthe 
send read pointer 456 is equal to the send write : poihter'454. - - - ' : 

Vdicribed a&ve'^ connection wi^Ste* SIX); tteCPU-310 generttes-a destination 
vectorV^catmg me director, ^ As also . : 

indicated above the command field' is 32-bytes, eighVbyte^^ereof havmg a bit representing a 
corresponding one of the 64 directors" to receive ^ message. For example,referring to FIG. 
UC each of the bit positions 1^64 represents directors 180,-18031. 200,-2003,, respectively. 
Here in this example, because a logic 1 is only in bit position 1, the eight-byte vector 
indicates that the destination diredor is only front-end director 108,. In the example m FIG. 
' UD because alogic 1 is only in bit portion 1, ^e'eightbyte vector indicates that the 
destination director is only fronted director « In tHe example in FIG. 1 IE, because a 
logic 1 is more than one bit position; the destban6h'for"the message is to more than one 
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director, i.e;, a multi-cast message. In- the; example in FIG. 1 IE, a logic 1 is only in bit . 
positions 2, 3, 63 and 64 ; Thus, the eight-byte vector indicates that, the destination directors 
are only front-end director 108 2 and 180 3 and back-end directors 200 3l and 200 32 . There is a 
mask vector stored in a register of register section 420 (FIG. 7) in the message engine (ME) 
315 which identifies director or directors ^hich may be not available to use (e.g. a defective 
director or.a director not in the system at that time), Step 524, 525„for a uni-cast 
transmission). If the message engine (ME) 315 state machine 410 indicates that the director 
As available by examining the transmit vector mask. (FJG r ,1 IF) stored, in register 420, the 
message engine (ME) 315 encapsulates the message payload with a MAC header and CRC 
inside the packetizer portion 428P, discussed above (Step 526). An example of the mask is 
shownii FIG, 1 IE. The mask has 64 bit positions, one for each one of the directors. Thus, 
as with the destination vectors described above in connection with FIGS. 1 1G-1 IE, bit 
positions 1-64 represents directors 180,-1 80 32 .200 l -200 32 , respectively. Here in this 
example, a logic i in a bit position in the mask indicates that the representative director is 
available and a logic 0 in such bit position indicates that the representative director is not 
available. Here, in the example shown in FIG. I IF, only director 2OO32 is unavailable. Thus, 
if the message has a destination vector as indicated in FIG. 1 IE, the destination vector, after 
passing through the mask of FIG. 1 IF modifies the destination vector to that shown in FIG. 
, 1 1G. Thus* director 2OO32 will not receive the message. Such mask modification to the 
destination vector is important, b^aus^as will be described, the messages on a multicast are 
sent sequentiaUy/andi^t .in par^lel. Jhus» ^M 111 ^**?? of message transmission to an 
unavailable director or directors mcr^ the message transnnssion efficiency of the system. 

- .-Having packetized ^ the message mtp a MAC packet^ 
packetizer/d-J-packetizer *428 (FIG, J), the message engine (ME) 315 transfers the MAC 
packet to the crossbar switch 320 (S^28) and the MAC packet is routed to the destination 
by the message.™^ 3041 ' 3042 oroiltlie 

same .director.board via the crossbar switch . 320 on such director board. 

Referring to FIG. 12, the message read operatic described. Thus, in Step 600 the 
director waits for a message. When a menage is received, me message engine (ME) 315 
state machine 410 receives the pac^^o^). ' "^state machine 410 checks the receive 
bit vector mask (FIG. 1 1 stored in register 426) against die source address of the packet (Step 
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604). If the state machine 410 determines that the message is from an improper source (i.e., 
a faulty director as indicated in the mask; FIG. 1 IF, for example), the packet is discarded 
(Step 606). On the other hand, if the state machine 410 determines that the packet is from a 
proper or valid director (i.e., source), the message engine (ME) 315 de-encapsulates the 
message from the packet (Step 608) in de-packetizer 428D. The state machine 410 in the 
message engine (ME) 315 initiates a 3 2-byte payload transfer via the DMA receive operation 
(Step 610). The DMA writes the 32 byte message to the memory receive queue 3 12R in the 
RAM 312 (Step 612^ The message engine (ME) 315 state machine 410 then increments the 
receive write pointer register 450 (Step 614):- The CPU 310 then checks whether the receive 
write pointer 450 is equal to the receive read pointer 452 (Step 616). If they are equal, such 
conditionindicates to the CPU 310 that a message ha^ riot been received (Step 618). On the 
other nand, if the receive write pointer 450 , ahd the receive read pointer '452 are not equal, 
such condition indicates to the CPU 3 10 that a message has been received and the CPU 3 10 
processes the message in the receive queue 3 14R of RAM 3 12 and then the CPU 310 
increments the receive read pointer and writes it into the receive read pointer register 452. 
Thus, messages are stored in the receive queue 312R of RAM 312 until the contents of the 
receive read pointer 452 and the contents of the receive write pointer 450, which are 
initialized to zero during power-up, are equal. 

Referring now to FIG. 13, the acknowledgement of a message operation is described. 
In Step 700 the receive DMA engine 420 "suice^sniUy completes a message transfer to the 
receive queue in RAM 312 (FIG. 7j/ fte statS&ne > 410 in the message engine (ME) 
' 315 generates an acknowledgem^^MAC packet arid transmits the MAC packeHo the - 

sendmgdkectorvia^e message^ 
: engine (ME) 315 atmesendmgdfcect^ 

' acknowledgement MAC pack^diw^rs^ payldad' via f a receive- DMA • 
operation (Step 706). The DMA of me sending (^source) director writes to a status field 
of the descriptor within. the RAM memory, send queue 314S (SKp 708). The state machine 
410 of the message engine (ME) 315 of the sendhig director (which received the' 
acknowledgement message) increments its/sendread pointer 454 (Step 712). The CPU 310 
of the sending director (which received me acknowledgement message) processes the 
descriptor status and removes the descriptor from the send queue 312S of RAM 312 (Step 
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714). It should be noted that .the sendand receive queues. 312S and 312R are each circular 

queues.'- • ci . .• ,•• ; : ■• ■ - • ■ • ■ 

As noted above, the MAC packets ate, during nqrmal operation, transmitted 
alternatively to one of the pair of message network. boards 304,, 304* by hardware a selector 
S in the crossbar switch 320.- The selector S is responsive to the bit B in the header of the 
MAC packet (FIG, 2B) and, when such bit B is one logic state the data is coupled to one of 
the message networks boards 402A and in response to the opposite logic state the data is 
coupled to the Other one of the message networks . boards 402B, That is, when one message 
is transmitted to board 304 t the next message is transmitted to board 3042. 

.Referring again to FIG. 9, the.details of an exemplary transmit DMA 418 is shown. 
As noted above, after a descriptor has been created by the CPU 310 (FIG. 7) and is then 
stored in the RAM 312. If the send writepointer. 450 (FIG. 7) and send readVpointer 452, 
described above, have different counts an indication is provided by the state machine 410 in 
the message engine (ME) 315 (FIG. 7) that the created descriptor is available for DMA 
transmission to the message engine (ME) 3 15, the payload off the descriptor is packetized 
into a MAC packet and sent through the message network 360 (FIG. 2) to one or more 
directors 180,-180 32 , 2e0 t -200 3 2. More particularly, the descriptor created by the CPU 310 is 
first stored in the local cache memory 3 19 and is later transferred to the send queue 3 12S in 
RAM 312. When the send write pointer 450 and send read pointer 452 have different counts, 
the message engine (ME) 3.1 5,state machine. 410 initiates a DMA transmission as discussed 
above in-connection withStep 520,^ 

m send queues 312Rwitbiff me : RAM3l£ Further, as noted above, each descriptor which 
contain* the message ^ 

is created by 4he CPU 3,10 5 it is sequentially stored to a sequential location, or address in the 
send.queue 3 12S. Here,, the address is a 32-bit address. 

. When the transmit DMA ^initiated," the state machine 410 in the message engine 
(ME) 315 (FIG. 7), sends the queue address on bus 411 to an address register 413 in the 
DMA transmitter 418 (FIG. 9) along.with.a : teansmit write enable signal Tx_WE signal. The 
DMA transmitter 4.18 revests the CPU bus 317 by asserting a signal on Xmit_Br. The CPU 
bus arbiter 414 (FIG. .7) performs a bus arbitration and when appropriate the arbiter 414 
grants the DMA transmitter 418 access to me CPU bus 3 17. the Xmit Cpu state machine 
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419 then places the address currently available in the address register 413 on the Address bus 
portion 3 17A of CPU bus 3 17 by loading the output address register 403. Odd parity is 
generated by a Parity generator 405 before loading the output address register 403. The 
address in register 403 is placed on the CPU bus 317 (FIG. 7) for RAM 312 send queue 
312S, along with appropriate read control signals via CPU bus 317 portion 317C. The data at . 
the address from the RAM 3 12 passes, via : the data bus portion 317D of CPU bus 317, 
tough a parity checker 415 to a data input register 417. The control signals from the CPU 
310 are fed to a Xmit CPU state machine "4 19 via CPU bus 317 bus portion 317C, One of the 
control signals indicates whether the most recent copy of the requested descriptor is in the 
send queue 312S of the RAM 312 or P resident in the local cache memory 319. That is, 
the most recent descriptor at any given address is first formed by the CPU 310 in the local 
cachememory 319 and islater transferred by the CPU 310 to the queue in the RAM 312. 
Thus there may be two descriptors with the same address; one in the RAM 312 and one m 
the local cache memory 319 (FIG. 7), the most recent one being in the local cache memory 
319 m eimer event, me 

transmission from the RAM 3 12 and this descriptor is stored in the transmit buffer register 
421 using signal 402 produced by the state machine 419 to load these registers 421. The 
control signal from the CPU 310 to the Xmit CPU state machine 419 indicates whether the 
m ost recent descriptor is in the local cache memory 319. If the most recent descriptor is m 
the local cache memory 319, the Xmit CPti Me mkchine 419 inhibits the data that was just 
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. read from send queue 312^ ^ 
" selector 423. msuchl:^ 
at the same address location The ^rc^im^^^^V the CPU 310 



passing to 



message m» machine 419 then ie-aAi.ra.es M«tPU bus SIT and after it is granted such. 
CPU bus 317.lh.Xmh CPU state n^iufie ^'uAeu reads ^ descriptor from the RAM 312. 
This time, however, there'the ,^^mPri«*Subl. »*» «d queue 312s » *. 
RAM 312 The descriptor ^mCittU^mUi^ the transmit buffer regtster 
421 in response to the assertion of the signal ioHyu&Xrnii CPU sate machine 419. The 
descriptor in the register 421 is then ^sfeftafeu^«iectt,r 423 to message bus interface 
409 under the control of a Xmit message (msg) Site machine 427: That is, the descriptor » 
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the transmit buffer register 421 is transferred to. th^ transmit, data buffer 422 (FIG. 7) over the 
32 bit transmit message bus interface 409 by-thermit message (msg) state machine 427. 
The data in the transmit data buffer : 422 (FIG. 7) is packetized by the packetizer section of 
the packetizer/d^packetizer 428 as described in Step 530 in FIG. 1 1. 

More particularly, and referring also to FIG.. 14A, the method of operating the 
transmit DMA 41 8 (FIG- 9) is shown. As noted above, each descriptor is 64-byte. Here, the 
transfer of the descriptor takes place over two -interfaces namely, the CPU bus 317 and the 
transmit message interface bus 409 (FIG. 7), The CPU bus 317 is 64 bits wide and eight, 64- 
bit double-words constitute a 64-byte descriptor. The Xmit CPU state machine 419 generates 
.the control signals which result in the transfer of the descriptor from the RAM 3 12 into the 
traiisrnit.buffer register 421. (FIG., 7). The.64-byte, descriptor is transferred in two 32-byte 
burst-accesses on the CPUiws 317. Each one of the eight double wordsis stored 
sequentially in the transmit buffer register 421 (FIG. 9). Thus, in Step 800, the message 
engine 315 state machine 410 loads the transmit DMA address register 413 with the address 
of the descriptor to be transmitted in the send queue 312S in RAM 312. This is done by the 
asserting the Tx^WE signal and this puts Xmit CPU state machine 419 in step 800, loads the 
address register 413 and proceeds to step 802. In step 802, The Xmit Cpu state machine 419 
loads the CPU transfer counter 43 1 (FIG. 9) with a 32-byte count, which is 2. This is the 
number of 32 byte transfers that would-be required to transfer the 64-byte descriptor, Step 
802. The Xmit Cpu state machme.41?.now proceeds to Step 804. In step 804, the transmit 
- DMA state machine 419 checks the^vajjdity ofthe address that is loaded into its address 
register 413, The address loaded into me address register 4 p is^ 

loadedinto me.memoryf .address register? 435 , The memory address registers 435 contain the 
base-address and the offset ofthe spnd q^eue 312 s ; in the RAM 312. The sum ofthe base 
address and the offset is^ejrange of { ad^esses for the send queue 312S in RAM 312. The 
address check circuitry 437 constantly checks whether the address in the address register 413 
is with in the range of the send queue 3 12S in the RAM 3 12. If the address is found to be 
: outside the range of me send queue^2S ^ status is stored in the 

. status register 404 and thgn passed back .to ,the message engine 315 state machine 410 in Step 
-4L6,The.checkforvalte^ 

i.e., valid, the transmit DMA sjate machine 4 19. proceeds with' the transfer and proceeds to 
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Step 806. In the step 806, the transmit DMA state machine 419 requests the CPU bus 3 17 by 
asserting the Xmit.BR signal to the arbiter 414 and then proceeds to Step 807. In Step 807, 
the Xmit'Cpu state machine 419 constantly checks if it has been granted . the bus by the 
arbiter. When the CPU bus 317 is granted; the Xmit CPU state machine. proceeds to Step 
808. In Step 808, the Xmit Cpu state machine 419 generates an address and a data cycle 
which essentially reads 32-bytes of the descriptor from the send queue 312S in the RAM 312 
into its transmit buffer register 421 .' The Xmit Cpu state machine 419 now proceeds to step 
810. In Step 810, the I Xmit Cpu state machine 419 loads the descriptor that was read into its 
buffer registers 421 and proceeds to Step 81 1. In Step 81 i, a check is made for any local 
cache memory 3 19 coherency errors (Le^checks whether the most recent data is in the Cache 
memory 319 and not in the RAM312) oh'these 32-bytes of data. If this data is detected^ be 
resident in the local CPU cache memory 319, then the Xmit Cpu state machine 419 discards 
this data and proceeds to Step 806. The Xmit Cpu state machine 419 now requests for the 
CPU bus 317 again and when granted, transfers another 32-bytes of data into the transmit 
buffer register 421 , by which time the CPU has already transferred the latest copy of the 
descriptor into the RAM 312. In cases when the 32-bytes of the descriptor initially fetched 
from the RAM 3 12 was not resident in the local CPU cache memory 3 1 9 (i.e., if no cache 
coherency errors were detected), the Xmit Cpu state machine 419 proceeds to Step 812. In 
Step 812, the Xmit CPU state machine 419 decrements counters 431 and increments the 
address register 413 so that such address register 4l 3 points to the next address. The Xmit 
Cpu state machine men proceeds^ step Sft' m Step 814; the Transmit CPU state 
machine 419 checks' to see if ^tt^ transfer counter' 431 has expired, i.e., counted*, zero, if the 
count was found to be non-zero^ itTthen, proceeds toStep 804 tostart the transfer of the next 
32-bytes of the descriptor. In case'thecount^r 4SB§ zero;the^rbcess^gbes to Step 816 to 
complete the transfer. The successful traksfe^tke second 32-bytes of descriptor from the 
RAM 312 into the transmit DMA buffer register' ^completes the transfer over the CPU 
bus 317. 

' ' The message mterface^ ^32* bits wide' and sixteen,- 32 bit words constitute, a 64- 
byte descriptor. The 64-jvte descriptor is ; trar*f*red in batches of 32 bytes each. The Xmit 
msg state machine 427 confro^ 

asserts the signal 433 to indicate that the first 32 bytes have been successfully transferred 
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over the GPU bus 317 (Step 818, FIG. 14B), this puts the Xmit msg state machine into Step 
818 and starts the transfer on the message interface. In step 820, the Xmit msg machine 427 
resets burst/transfer counters 439 and initiates the transfer over the message interface 409. In 
Step 820, the transferis initiated over the message interface 409 by asserting the "transfer 
valid" (TX_DATA_Vaild) signal indicating to the message engine 3 15 state machine 410 
that valid data is available on the bus 409, The transmit msg machine 427 transfers 32 bits of 
data on every subsequent clock until its burst counter in burst/transfer counter 439 reaches a 
: V alue equal to eight, Step 822. The burst counter in burst/transfer counter 439 is incremented 
with each 32-bit word put on the message bus 409 by a signal on line 433. When the burst 
countis eight, a check is made by the state machine 427 as. to whether the transmit counter 
43 J has expired, i.e., is zero, Step 824. . The expiry of the transfer counter in burst/transfer 
counter 439 indicates the 64 byte descriptor has been transferred to theTransmit buffer 422 in 
message engine 315. If it has expired, the transmit message state machine 427 proceeds to 
Step 826. In step 826, the Xmit msg state machine asserts the output End of Transfer 
(TxJEOT) indicating the end of transfer over the message bus 409 Jn this state, after the 
assertion of the Tx_EOT signal the status of the transfer captured in the status register 404 is 
sent to the message engine 315 state machine 410. The DMA operation is complete with the 
descriptor being stored in the transmit buffer 422 (FIG. 7). 

On the other hand, if the transfer counter in burst/transfer counter 439 has not 
• expired, the process goes to Step 800 and.repeats the above described procedure to transfer 

the 2 nd 32 bytes of descriptor data, at which time ^ti^et^beaimplete. 
r> v av Referring nowto FIG, 10, the receive DMA^Ois shown. Here, a message received 
••^m^cther.dtectotM torbe wttea into the RAM 312 (FIG..7). The receive DMA 420 is 
... adapted to handle three .types of mforrnation: error information which is 8 bytes in size; 
acknowledgement tafonn^qn,wbi^fe-,16 bytes in size; and receive message payload and/or 
fabric management information which is 32 byes in size. Referring also to FIG. 7, the 
message engine 315 state machine ..410 asserts the Rx^WE signal, indicating to the Receive 

DM A 420 that it is.ready transfer : the : Data in its Rec buffer 426 FIG. 7. The data in the 

. Receive buffer could be.the 8,byte error, information, the 16-byte Acknowledgment 
information or the 32.byTe Fabric management/ Receive ^ » . 

places a 2 bit encoded receive .transfer count, on the ^transfer count signal indicating the 
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type of information and an address ^hidh is the address where, this information is to be stored 
in the receive queue of RAM 312. Irrfespbnse to the receive write enable signal Rx_WE, the 
Receive message machine 450 (FIG. lO^loads the address into the address register 452 and 
the transfer count indicating the type of information, into the receive transfer counter. 454. 
The address loaded into the address register 452 is checked by the address check circuitry . 
456 to see if it is with in the range of the Receive queue addresses, in the RAM 312. This is 
done by checking the address against the Values loaded into the memory registers .457 (Le., a 
base address register and an offset register therein): The base address register contains the 
start address of the receive queue 3 12R ; residing in the RAM 3 12 and the offset register, 
contains the sizeof this receive queue 312R in RAM 312. Therefore the additive sumof, the , 
values stored in the base address register and the offset register specifies the range of 
addresses of the receive queue in the RAM 312R, The memory registers 457_jre loaded 
during mitiaUzatioh. On the subsequent clock after the assertion of the Rx_WE signal, the 
message engine 315 state machine 410 the proceeds to place the data on a 32-bit message 
engine 315 data bus 407, FIG. 10. A Rx_data_valid signal accompanies each 32 bits of data, 
indicating that the data on the message engine data bus 407 is valid. In response to.this 
Rx_data_vaUd signal the receive message state machine 450 loads the data on the data bus 
into the receive buffer register 460. The end of the transfer over the message engine data bus 
407d is indicated by the assertion of the Rx.EOT signal at which time the Receive message 
state machine 450 loads the last 32 bits of data on the message engine data bus 407IX of bus 
407 into theVeceive buffer registers 460; Tm^signals the end of the transfer- over the. 
"' menage engine^ buS'^TDMbn of 'bus^ At the end-ofsuch transfer is conveyed to 
"me RxjCpu state machine 4^2 by the assertion of the signal 464. -The. Receive CPUmachme 
462 now requests for the CPU bus 317 by asserting" the signal REG^Br. After an. arbitration 
" by CPU bus arbiter 414 (FIG. 7) the receive DMA 420 (FIG. 10) is given access.to the CPU 
bus 317. The Receive CPU state machine ^proceeds to transfer the data in its buffer 
registers 424 over the CPU bus 3 1 7 into theReceive queue 312R in the RAM 312.. 
Simultaneously, this data Wals6 tra^feircd ! m^a du^lic^ buffer register 466., The data at 
the output of the receive buffer register 460 passes to one input of a selector 470 and also 
passes to a duplicate datareceive buffer registef460. The output of the duplicate receive 
buffer register 466 is fed to a second input of the selecW470: As the data is being 
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transferred by the Receive CPU state machine 462, it is also checked for cache coherency 
errors. If the data corresponding to the address being written into the RAM 312, is located in 
the CPU's local cache memory 3 19 (FIG. 7), the receive DMA machine 420 waits for the 
CPU 310 to copy the old data in its local cache memory 319 back to the receive queue 312R 
in the RAM 312 and then overwrites this old data with a copy of the new data from the 
duplicate buffer register 466. 

More particularly, jf central processmg unit 310 indicates to 
that the data the. receive buffer register 460 is syllable in. the local cache memory 319, the 
receive CPU state machine 462 produces a select signal on line 463 which coupies the data in 
the duplicate buffer register 466 to meoutput of selector 470 andthen to the bus 317 for store 
in the randdin access memory 312. , r _ . 

The successful write into the RAM 312 completes the DMA transfer. The'receive DMA 420 
then signals the . message engine 315 state machine 410 on the status of the transfer! The 
status of the. transfer is captured in the status register 459. 

Thus, with both the receive DMA and the transmit DMA, there is a checking of the 
local cache memory 319 to determine whether it has "old" data, in the case of the receive 
DMA or whether it has "new data" in the case of the transmit DMA. 

Referring now to FIG. 15A, the operation of the receive DMA 420 is shown. Thus, in 
Step 830 the Receive message machine.450 checks if the write enable signal Rx_WE is 

- asserted. If found asserted, the receive DMA 420 proceeds to load the address register 452 
and the trarisfer ; counter : 454. The value, loaded into fce transfer counter 454 determines the 
type.of DMA transfer requested by th<? Message enguie state machine 310 in FIG 7. The 
assertioh^fme Rx.W^ signal starts ^ operation. This puts the Rx 
trisg state machine ; 450 in Step 83^ ; In Step ; 832 theRec msg state machine 450 loads the 
address register 45.2, the transfer counter 454 and then proceeds to Step 834. In Step 834, it 

. checks to. see if the Rx_D AT A_V A^DD signal is asserted If asserted it proceeds to step 836. 
The Rx msg statemachine loads the buffer . register^ O'lG. 10) in Step 836 with the data 
• on themessage engine data bus 407D ^^H^-^^S^^^^ 
accompanies each piece o/dat* put m ,S« bw 407. The data is sequentially loaded into the 

- buffer registers 460 (FICU 0). The End of the transfer^on the message engine data bus 407D 
of bus 407 is indicated by the assertion of the RxJEOT signal. When the Receive message 
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state machine 450 is in the End of transfer state Step 840 it signals the Receive GPU state 
machine 462 and this starts the transfer on the CPU bus 3 17 side. 

The flow for the Receive CPU state machine is explained below. Thus, referring to 
FIG. 15B, the End of the transfer oh the Message engine data bus 407D portion of bus 407 
starts the Receive CPU state machine 462 and puts it in Step 842. TheReceiVe CPU state 
machine 462 checks for validity of the address in this state (Step 844). This is done by the 
address check circuitry 456. If the address loaded in the address register 452 is outside the 
range of the receive queue 312R in the RAM 312, the transfer is aborted and the status is^ 
captured in the Receive status register;459 and the Rec Cpu state machine 462 proceeds to 
Step 845. On a valid address the Receive CPU state machine 462 goes to Step 846. In Step 
846 the Receive Cpu state machine 462 requests for access of the CPU bus 317. It then 
proceeds to Step. 848. In step 848 it checks for a grant on the bus 317. On" a.qualified grant it 
proceeds to step 850. In Step 850, The Rec Cpu state machine 462 performs an address and a 
data cycle, which essentially writes the data in the buffer registers 460 into the receive queue 
312R in RAM 3 12. Simultaneously with the write to the RAM 3 12, the data put on the CPU 
bus 3 17 is also loaded into the duplicate buffer register 466. At same.time, the CPU 310 also 
indicates on one of the control lines, if the data corresponding to the address written to in the 
RAM 312 is available in its local cache memory 319. At the end of the address and data 
cycle the Rec Cpu state machine 462 proceeds to Step 850. m this step it checks for cache 
coherency errors of the type described above ^connection with me transmit DMA 418 (FIG. 
9) If cache coherency error is'detectWan* &e receive CPU state machine 462 proceeds to 
' Step 846 and retries the uinsac^nlnore parfcularry, the Receive CPU state machine 462 
now generates ^ arid^his time the data 

from the duplicate buffer 466 is put on to the CPU data bus 317: If there were no cache 
coherency errors the Receive CPU state machine 4^ Vroceeds to Step 852 where it 
decrements the transfer counter 454 and mcremerlt^e address "in the address register 452. 
. The Receive Cpu state machine 462 then proceeds to Step 854. In Step 854, the state 
machine 462 checks if the frar^fer counter ha^ expired; i.e., is zero. On a non zero transfer 
count the receive Cpu staje machine 462 proceeds to Step 844 and repeats the above 
described procedure until the transfer becomes '& A zero transfer count when in step 854 
completes the write into the receive queue 3 1 2R in RAM 312 and the Rec Cpu state machine 
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proceeds to 845. In step 845, it conveys status stored in the status register back to status is 
conveyed to the message engine 315 state machine 410. 

Referring again to FIG. 7, the interrupt control status register 412 will be described in 
more detail. As described above, a packet is sent by the pocketsize portion of the 
packetizer/de-packetizer 428 to the crossbar switch 320 for transmission to one or more of 
the directors. It is to be noted that the packet .sent by the packetizer portion of the 
packetizer/de-packetizer 428 passes through a parity generator PG in the message engine 315 
prior to passing to the crossbar switch 320. , When such packet is sent. by the message engine 
315 in exemplary director 180,, to me crossbar switch 320, a parity bit is added to the packet 
by parity bit generator PG prior to passing to the crossbar switch 320. The parity of the 
packet-is checked in the parity checker portion of a parity checker/generator (PG/C) in the 
crossbar switch 320. the result of the check is sent by the PG/C in the crosstrar switch 320 
to the interrupt control status register 412 in the director 180i. 

Likewise, when a packet is transmitted from the crossbar switch 320 to the message 
engine 3 1 5 of exemplary director 1 80, , the packet passes through a parity generator portion 
of the parity checker/generator (PG/C) in the crossbar switch 320 prior to being transmitted 
to the message engine 315 in director 180,. The parity of the packet is then checked in the 
parity checker portion of the parity checker (PC) in director 180, and is the result (i.e., status) 
is transmitted to me statxis register 412:-> ; . , 

A number of embodiments of the invention have been described. Nevertheless, it will 
be understood that various modifications may be made without departing from the spirit and 
scope of the mvehuW Accordingly, other embodiments are within, the scope of the 
following claims. 
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CLAIMS - - -: -. 

1. A system interface comprising: 
a plurality of first directors; 
a plurality of second directors; 

' a data transfer section -hiving a cache memory, such cache memory being 
coupled to mWpluraUty of first and second directors; 

: - a messaging network, operative independently of the data transfer section, 
coupled tothe plurality of first directors and the plurality of second directors;. and 
wherein the fcV and "second directors control data transfer between the first 
directors and the second directors in response to messages passing between the first 
directors and the second directors throughthe- messaging network to facilitate data. 
- transfer between first directors and the second directors with such ^.passing through 
me cache memory in the data transfer section. 



2. The system 
includes: 



interface recited in claim 1 wherein each one of the first directors 

a data pipe coupled between an input of such one of the first directors and the 

cache memory; 

a controller for .referring the messages between the message network and such one 



of the first directors. 



svs^m iu^facc'recited in Claim l or claim 2 wherein each^ne of the second 



3 . ' ' The system' 

directors includes: ~: ■ 

a data pipe coupled between an input of such one of the second directors and 

the cache memory; 

a controller for transferring the messages between the message network and such one 

of the second directors. 

4. The system interface recited in claim 2 wherein each one of the second 
directors includes: ^ . 
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a data pipe coupled between an. input of such one of the second directors and 
the cache memory; 

a controller for transferring the messages between the message network and such one 
of the second directors. 

5. , • .. The system interface recited in any preceding claim wherein each one of the first 
directors includes: 

a data pipe coupled between an input of such one of the first directors and the 

cache memory; ' . - : : • . .- 

j& microprocessor; and . . ; . : 

a controller coupled to the microprocessor and the data pipe for controlling the 
transfer of the messages between the message network and such one ofthe-first directors and 
for controlling the data between the input of such one of the first directors and the cache 
memory. 

6. -The system interface recited in any preceding claim wherein each one of the second 

directors includes: . , 

a data pipe coupled between an input of such one of the second directors and 

the cache memory; 

a microprocessor; and , . ,. V4 .-. .„ 

a.controller coupled to the microprocessor and the data pipe for controlling the 
-' transfer of the messages between the rnessage network and such one of the second directors 
and for controlling the data between the input of such one^fm^secohd directors and the 
cache memory. v,\ _ ; • :/: v ; - , ^, 

7. The system interface recited in claim 5 wherein each one of the second 

directors includes: ........ 

a data pipe coupled between an input of such one of the second directors and 

the cache memory; . , . 
a microprocessor, and 
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transfer of the messages between the message network and such one of .he second director, 
„<, for controlling the da» between me input of such one of*e second directors and the 



cache memory. 
8 



e A data storage system for transferring data between a host computer/server ; 
and a bank of disk drives through a system interface, such system interface comprising: , 
a plurality of first directors coupled 'to host computer/server, 
a plurality of second directors coupled to the bank of disk drives; 
a data transfer section having a cache memory, such cache memory being coupled to 
the plurality of first and second directors; ~ 
- a messaging network, operative tadependeritly'bf the data transfer^ection, coupled to 
the plurality of first directors and the plurality of second directors; and 
wherein the first and second directors control data transfer between the host computer 
and .he bank of disk drives in response to messages passing between the first directors 
and me second direcmrs through me messagir^netwo*^ 
betweenhost computer/server and thebank of disk drives with such daa passing 
through the cache memory in the data transfer section. 

o The system interface recited in claim 8 wherein each one of the first directors 

" includes: , A . u „ 

mCm0 ^ controller for transferring the messages between the message network and such one 

of the first directors, ....... f . 



l0 . The system interface recited. in claim 8 or claim 9 wherein each one of the second 

7 directors includes: ^ 

a data pipe coupled between an input of such one of the second doctors and 



4 the cache memory; 
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5 a controller for transferring the messages between the message network and such one * 

6 of the second directors. 

1 11. The system interface recited in claim 9 wherein each one of the second 

2 directors includes: 

3 a data pipe coupled between an input of such one of the second directors and 

4 the cache memory; 

5 t ■ v .a controller for transferring the messages between the message network and such one 

6 of the second directors. 

1 ;/: i2:^_ t The system interface, recited .in any. of .claims. 8 to 11 wherein each one of the first 

-2 directors includes: " — 

3 a data pipe coupled between an input of such one of the first directors and the 

4 cache memory; 

5 a microprocessor; and 

6 a controller coupled to the microprocessor and the data pipe for controlling the 

7 transfer of the messages between the message network and such one of the first directors and 

8 for controlling the data between the input of such one of the first directors and the cache 

9 memory. , 

13. The system interface recited in any j of claims 8 to 1 2 wherein each one of the second 

directors includes: ' " ' i ' : • 1 f ' r ";^f : ■ * .'■ •;■ ■ 

2 ' 

3 . N a data pipe coupled between an input of such one of the second directors and 
4, the cache memory; _ : , 

5 a microprocessor; and t 

6 a controller coupled to the microprocessor and the data pipe for controlling the 

7 transfer of the messages between the message network and such one of the second directors 

8 and for controlling the data between the input of such one of the second directors and the 

9 cache memory. 



40 



14. The system interface recited in claim 12 wherein each one of the second 
directors includes: 

a datapipe coupled between an input of such one of the second directors and 

the cache memory; 

a microprocessor; and 

a controller coupled to the microprocessor and the data pipe for controlling the 
transfer of the messages between the message network and such one of the second directors 
and for controlling the data between the input of such one of the second directors and the - 

cache memory. 

15. Amethod for operating a data storage system adapted to transfer data between 
ahosfcomputer/server andabank of disk drives, such method comprising: - 

transferring messages through a messaging network with the data being 
transferred between the host computer/server and the bank of disk drives through a 
cache memory, such message network being independent of the cache memory. 

16 A method of operating a data storage system adapted to transfer data between 
a host computer/server and a bank of disk drives through a system interface, interface 
comprising: a plurality of first directors coupled to host computer/server, aplurauty of 
second directorscoupled to the bank of disk 

memory, such cache memory being coupled to the plurality of first and second directors, 

such method comprising:. .• : , . .... .. . - u , 

transferring the databetween the host computer/server and the bank of disk 
drives under control of the first and second directors in response to messages passing 
between the first directors.and the second directors through a messaging network to 

faciUtatethedatatra^^ 

\ suchdaupassmgthr^ 

network being independent of the cache memory. 
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1 17. A method of operating a system interface having a plurality of first directors, a 

2 plurality of second directors and a data transfer section having a cache memory, such cache 

3 memory being coupled to the plurality of first and second directors, such method comprising: 

4 providing a messaging network, operative independently of the data transfer 

5 section, coupled to the plurality of first directors and the plurality of second directors 

6 to control data transfer between the first directors and the second directors in response to 

7 ' messages passing between the first directors and the second directors through the 

8 messaging network to facilitate data transfer between first directors and the second 

9 directors with such data passing through the cache memory in the data transfer section. 

1 .^18. The method recited in claim .17 ..including pro viding each one of the first 

•2 directors is provided with: " 

3 a data pipe coupled between an input of such one of the first directors and the 

4 cache memory; 

5 a controller for transferring the messages between the message network and such one 

6 of the first directors. 

1 19. The method recited in claim 17 or claim 18 including providing each one of the second 

2 directors with: 

3 a data pipe coupled between an input of such one of the second directors and 

4 ' me cache memory; ■ •'■'.= <'■ ."• ra ;■, .. a ; 

5 a controller for transferring the messages between the message network-and such one 

6 of the: second directors. , ; ; Ls .: ? • . : . < 

1 20. The method recited in claim 1 8 including providing each one of the second 

2 directors with: < : ' ; " -• "«~ .;. 

3^ 'a d^pipecoupMbe^eOT an.u^utof s^ch one of the second directors and 

4 ' the'cactie memory; * • ■ ' --'^v >o in-.--:' ; . . . ^ , 

5 a controller for transferring the messages between the message network and such on 

6 of the second directors. ^ 



21. The method recited in cltol-7ancluding ; providingeach oneof the first 

directors with: J ' ' . 

a data pipe coupled between an input of-such.pne of the first directors and the 

cache memory; v 1 

a microprocessor, and - % 

a controller coup ted to the nucrtprocessor and the data pipe for controlling the 
transfer of the messages between fctmessage network and such one.of the firs, doctors and_ 
for controlling the databetweeStte u^of such one of*, first directors and the cache 
memory. 

22. The method recited in any o? claims lT to:2riiiciuding providing each one of the 

secondLdirectors with: - . - - - — - 

a data pipe coupled between an input of such-one of the second directors and 

the cache memory; 

a microprocessor; and 

a controller coupled to the microprocessor and the data pipe for controlling the 

and for controffing the databetween the input of such one of *e second directors and the 

cache memory. 

"23. The method recited in claim 21 including providing each one of the second 
directors with: 



upled between an input of such one of ^second directors and 



a data pipe co 

the cache memory; 

a microprocessor; arid 

a controller coupled to the microprocessor and the data pipe for controlling the 
nansfer of*, messages hereto the messa-ge— and such oneof the second directors 
and for controlling the data between the input of such one of the second directors and the 
cache memory. 

24. A system interface as hereinbefore described with reference to figures 2 to 15B. 

25. A data storage system as hereinbefore described with reference to figures 2 to 15B. 

26. A method as hereinbefore, described with reference to figures 2 to 15B. 
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Amltom nts to the claims hav b d a follows 



27. A data storage system for transferring data between a host computer/server and a bank of disk ^ 
drives through a system interface, such system interface comprising: 

a plurality of first directors coupled to the host computer/server; -- 
a plurality of second.directors coupled to the bank of disk drives; 
a cache memory; and 

^ofo rmrhq between each one of the directors and 
.- wherein there are separate point-to-point data patns Detween cat-u u 

the global cache memory. 

28. The system recited 'in claim 27 including a backplane and wherein the cache memory and the 
directors are interconnected through the backplane, -.' 

29. The system recited in claim 28 whereinthe backplane is a printed circuit board. 

30. The system recited in claim 27 further comprising: 

• i j 1 *u „i„«ii;tv of first directors, the second directors, and 
a data transfer section coupled to the plurality or nrsx airc^i , 

the cache memory, and , 
a messaging network coupled to the plurality of first directors and the plurahty of second 
directors such first and second directors controlling data transfer between the host computer and 
the bank of disk drives in response to messages passing between the directors through the 
. messaging network as such data passes through the memory via the data transfer section. 

" 31. The system recited in claim 30 includmg a backplane and wherein- the cache memory and the 
directors are interconnected through the backplane. .. : . . 

32. The system recited in claim 31 ' te^^* * - 

33 A nxethod for transferring data bTcween a host compuWr/server and a bank of disk drives 
. touch a system hiterface, such system interface comprising: a plurality of first directors 

c - * * '"-r.- , .I-* A r c . rn nH directors coupled to the bank of disk: 

coupled to the host computer/server, a plurality of second directors p 

, ^ - u i lrt iifv nf first directors and second directors; ana 
drives; a data transfer section coupled to the plurahty.oi tirst qirec u 

« * 0 f nr - r directors and the plurality of second 

a messaging network coupled to the plurality of first directors ; h j 

., ino Hpfja transfer between the host computer and 
directors, such first and second directors controlling data nransier o 
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***\ n a herween the directors ihrough the 
the bank of disk drives in response to messages passing between 

l u Hnra transfer section, such method comDnsing: 
messaeino network as such data passes ihrough the data transrer 

15 = f . j= , messace to be sent to a receiving one, or 

preparing in a transmitting one of the directors, a message 

ones of the directors; 

j- _»__- a descriptor, such descriptor comprising a 
huildin- in such transmitting one of the directors, a descnpco, y 

• • • „ n „, or ones of directors to receive the built message; 

command field indicating the rece.vmg one, or ones, 01 

i ^r'tin,. nescriDtor into a packet, such packet composing: 
encapsulating the message payloadof the descriptor im h 

a header, such header including: 

a source portion indicating the transmitting one of the directors; . 

- • A „u na the receiving one, or ones, of the directors; and 
a destination portion indicating the receiving 

the message payload; , . 

. ...... . or on es of the directors through the 

transmitting such packet to said receiving one, or one-, 01 ... . .. 

messaging network; _ receiving one, or 

decoding the destination portion of the packet to route sucn pac* 

ones of the directors. 

34 The method recited in claim 33 including: 

• - « or ones of the directors the transmitted packet; 
receiving in one of the receiving one or ones oi me oix ...... . . 

receiving receiving directors whether the received 

determining in such receiving one,.orones, the receiving 

w from a proper or an improper transmitting one of the directors; 
packet is from a proper, or P P of ^ Sectors the received packet after 

de-encapsulating in such receiving one, or ones, 

--• ■ •• • • • . ... .•<-,.--■ = f . tl k directors that the.packet is. from a proper 
determining in such receiving one, or ones, of the directors y 

transmitting one of the directors. 

t. • „ receiving one, or ones, of the directors 

35 The method recited in claim deluding havinj. the receding , 

rr^n^mittin^ one of the transmitting such 
send an acknowledge receipt of the packet to said transmit^ 

packet. 

• - u -^ir,., in the descriptor in such transmitting one of the 

The method of claim 33 where-in?in;building, in the aes v 

36. The metno ind icating an address m the bank of 

directors, the descriptor further comprises a messa = e ; pay 

disk drives having the requested data. . . . 
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37 The method for transferring data between a host computer/server and a bank of through a , 

• • • „• -i nlnralitv of first directors coupled to the 
system interface, such system interface comprising, a plurality or 

r a a' rritmled to the bank of disk drives; a data 

"host computer/server; a plurality of second directors coupled m 

transfer section coupled to the plasty of first directors and second directors; and a messaging 
^twork coupled to the plurality of first directors and the plurality of second directors, such first 
and second directors controlling data transfer between the host computer and the bank of disk 
drives in -response to messages passing between the directors through the messaging nerwork as 
such-data passes through the data transfer section," such method comprising: 

. - „ nMn f t Uf, "directors that "action is requested by a receiving one, 
determining, m a transmitting one of the directors, w 

or ones, of the directors; 

• • f*u„ Airrrtnr* in response to such determination, a 

preparing in such transmitting one of the directors, m respu 

message to-be sent to the receiving one, or ones, of the directors; . . . 

, " ' ' '"''StL Ht'rectois a descriptor* such descriptor comprising a 

building in such transrrumng one of the directors, a F 

- HiTectors to receive the built message; 

comm-and field indicating receiving one, or ones, of the directors to rece 

n „, „j t Mn such transmitting one of the directors; 

storing the built descriptor in a memory within sucn nau* 

• . * ^„r-rr,;*h-ncr one of the directors each time a 
incrementing a pointer or counter in the transmittmg one or m ^ 

. , ^ . nrv :„ srnch transmitting one of the directors; 

descriptor is ready to be retrieved from the memory in such transmi s 

a piu ^_ t , _ mm -v in such transiting one of the directors; 

retrieving such stored descriptor from the memory in sucn trans 

• v i »A nf the retrieved descriptor into a MAC type packet, such 

encapsulating the message payload of the retnevea u<= v 

packet comprising: 

. a header, such header mcluding: 

. . a source portion indicating the one or uie ur<^ & 
'" ' ' - J :-i - - oWA^irone or ones of me receiving .directors, 

a destination portion indicating the anoJiwr one, or on . . 

and 

: the message payload; , 
. ^tting such packet to said}»one, or ones, of the directors through the 

messaging network; _ ^ ^ ' ^ ^ ^ QT 

decoding the destination portion of the packet to route su^ F 

ones, of the directors.. 

38 The method recited in claim 3? including: ■■■ 

• • ~roneq nf the directors the transmitted packet; 

receiving in one of the receiving one, or ones, or me a 



determine in such one of .he receiv.ng one. or ones, of the directors whether U» 
rec -ived packet is from a proper, or an .mproper transmitting one of «he d.rectors; 

'■' • ^r^n^c nf the directors the received packet 

de-encapsulating in such one of the receiving one, or ones, of the dire 

nr nnes of the directors that the packet is 
after determining in such one of the receiving one, or ones, 

from a proper transmitting one of the .directors, 

, 9 ^ e method recited in claun 38 including having the receiving one, or ones, of the directors 
send an acknowledge receipt of- the packet to said transmitting one of the directors . transmmmg 

such packet. 

40 Tie method recited in claim 39 including incrementing in said transmitting one of the 
directors pointer or counter as a consequence of rM^^^^' 

41 The method recited in claim 37 wherein the descriptor incudes a message payload 
indicating an address in the bank of disk drive, having the requested data. . - 

42. The method recited in claim 41 including: 

' . . ~* n ** of the directors the transmitted packet; 

receiving in one of the receiving one, or ones, oi tne oircc 

detennining in such one of the receiving one, or ones, of the directors whether the 

^caJulating in such one of the., receiving «g« ones, of the' directors the received jacket 
de-encapsuiams ... . : , ^ directors that the packet is - 

after determining in such one of.the receiving one, or ones, ot tne oir 

from a proper transmitting one of the directors. 

t. - „ rtv receiving one, or ones, of the directors 
43 Tae method recited in claim 42 including hayuig the receiving o 

, '•V^nc'mittine one Of the directors transmitting 

send an acknowledge receipt of the packet to said transmitting one 

such packet. ••- : , , c . 

„ Tne method recited in data 43 indud*g incrementing m said transmitting one 
directors a pointer or counter as a cons=, u ence,o£ receiving such acknowledge rece.pt. 

: «^«MM»r/«;erver and a bank of through a 
45 A method for transferring data between a host computer/server ana 

• • a ninrplitv of first directors coupled to me 
system interface, such system interface composing, a plurality 
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host computer/server; a plurality of second directors" coupled to the bank of disk drives; a data, 
transfer section coupled to the plurality of f.rst directors and the second directors; and a 
messaging network coupled to the plurality of first directors and the plurality of second directors, 
such first and second directors controlling data transfer between the host computer and the bank of 
disk drives in response to messages passing between the directors through the messaging network as 
such data passes through the data transfer section, such method comprising: 
.determining, in a transmitting one of the directors, that data requested for transfer by such 
transmitting one of the directors is unavailable in the cache memory; 

preparing in such transmitting one of the directors, iri response to such determination, a 
message to be. sent to a receiving one, or ones, of the directors; 

,buildingjn such transmitting one of the transmitting .Sectors, * descriptor, such descriptor 

14 comprising: 

a message payload indicating an address in the bank of disk drives having the requested 
16 4?+*] and 

a command field indicating receiving one, or ones, of the directors to receive the built 

is message; - ~ . • 

storing the built descriptor in a memory within such transmitting one of the directors; 
incrementing a poinfer or counter in the trarisrmtting one of the directors each time a 
descriptor is ready to be retrieved from the memory in such transmitting one of the directors; 
retrieving such stored descriptor from the memory in such transmitting one of the directors; 
decrementing the pointer orcounter when the descriptor is retrieved from the memory m 
such transmitting one of the directors; ^ „ 

..encapsulating the message payload of the retrieved descriptor into a MAC type packet, such 

26 packet comprising: 

27 a header, such header including: 
a source portion indicating the one of mb'transmitting directors; " 
a destination portion indicating the another one, or ones of the receiving directors; 
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: the message payload; .... _ 
transmitting such packet to said recei ving one, or ones, of the' directors through the 



31 
32 

33 messaging network; 

decoding the destination P omon : of the packet to route such packet to such receiving one, or 

35 ones, of the directors. 
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46. The method recited in claim 45 including: 

- • 'ex. • nmhes of the directors the transmitted packet; 

receiving in one of the receiving one, or ones, oi r 

' determining in such one of the receiving one," or ones, of-the directors whether the 
received packet is from a proper, or an improper transmitting one'bf the directors; 
de-encapsulating in such one of the receiving one, or ones, of the directors the received packet 

i c u nn? nrones of the directors that the Dacket is 

after determining in such one of the receiving one, oi ones, ^ 

from a proper transmitting one of the directors. 

47 The method recited in.clata 46 inCuiing having the receiving one ; o, ones, of the doctors 
xaA an acknowiedge receipt of the packet to said chitting one of the directors transm.cung 

3 such packet : ^ 

, 48 The method recited in claim 47 including incremennBg in said tran^nittmg one of the : 
directors a pointer or counter as a consequence of receiveing such acknowledge receipt. 

49 The method recited in claim 45 wherein the descriptor deludes a message payload 
Seating an address In the bank of disk drives having the requesteddata, 

50 The method recited in claim 49 including: 
. 1' • " ' • •„ „^ ; or ones °6 ; fthe directors the transmitted packet; 

receiving in one of the receiving one, or ones, oi uic 

.. -detemumngmsuchone.ofthcrecei^^ ~ 
reived packet b from a proper, orantopropeV^riaWtdng one of ^directors; 

, nrones of the directors' the received packet 

de-encapsulating in such one of the receiving one, or ones, oim 

, • r!i-«T nnc»^or~ones' 'of the directors that the packet is 

after determining in such one of; the receiving ; one, or ones, oi in 

from a proper transmitting one of the directors/ 
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, 51 The method recited in claim 50 including having the receiving one, or ones, of the directors 

, a ^ncrriittinc' one of the directors transmitting 

2 send an acknowledge receipt of .the packet to said lxansmittm 0 one o 

3 such packet. 

. , j- - -^'^pnrino 5 in said transmitting one of the 
; 52. The method recited in claim 51 including increment 

nf receiving such acknowledge receipt. 
2 directors a pointer or counter as a consequence ot receive 

49 
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53. The method recited in claim 33 wherein the.messages pass between the directors through the 
messaging network as said data passes through the cache memory via the data transfer section. 
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